Quickies for Unix

Borkware »
Quickies »
Quickies for Unix

Administration

Adding a new machine alias via netinfo(e) [permalink]
The NetInfo Manager utility is painfully slow. Here's how to add a machine via the command line. In this case adding the IP for a machine called "tower" (thanks to rzolf)
```
# niutil -create . /machines/tower
# niutil -createprop . /machines/tower ip_address 10.0.1.155
# niutil niutil -createprop . /machines/tower serves ./local
```
Increasing shared memory limits [permalink]
Edit /System/Library/StartupItems/SystemTuning/SystemTuning
tweak some values, like
```
sysctl -w kern.sysv.shmmax=167772160 # bytes: 160 megs
sysctl -w kern.sysv.shmmin=1
sysctl -w kern.sysv.shmmni=32
sysctl -w kern.sysv.shmseg=8
sysctl -w kern.sysv.shmall=65536 # 4k pages: 256 megs
```
Save, and restart your system.
On some Panther seeds, you may need to add these things to /etc/rc
Updating critical system files while preserving the original timestamp [permalink]
Sometimes you need to edit a file, like /etc/rc, but you want to be able to back out your changes. You also want to preserve the original timestamp of the file. That way if you back out your change, someone else coming along doesn't have to figure out "why did /etc/rc change yesterday, it doesn't look any different?" Here's how:
1. Move (not copy) the original to a backup name. This will preserve the timestamp: % sudo mv rc rc-orig
2. Copy the original to the usual name: % sudo cp rc-orig rc
3. make your edits: % sudo vi rc
If you decide your changes aren't worth keeping, you would move the original back:
% sudo mv rc-orig rc
Thereby undoing your changes, and not messing up the timestamp of the original file.
(mucho thankos to Louis Bertrand for this one)

General

Backups on the cheap [permalink]
I've been using hfstar for doing backups. I tar directory hierarchies to an external firewire drive from my TiBook (say my entire $HOME directory, plus the /Applications directory. There's probably other stuff that should be snarfed (like /Library), but I didn't on my last backup, and doing a restore from scratch seemed to work OK. Here's a quick outline of backing up to the external firewire drive called "Wikkit":
- % sudo mkdir /Volumes/Wikkit/backup
- % cd /Users
- % sudo hfstar cf - bork | (cd /Volumes/Wikkit/backup; sudo hfstar xf -)
The tar above is the standard push-pull tar. Both sides are run as root so that owner, group, and permissions are preserved for the files.
To Restore:
- % cd /Users
- % sudo mv bork bork-orig
- % cd /Volumes/Wikkit/backup
- % sudo hfstar cf - bork | (cd /Users; sudo hfstar xf -)
Some folks have had success with psync, but we haven't used it yet.
Case-insensitive file completion with bash [permalink]
If you use bash as your shell and use filename completion a lot, you can get bash to ignore the case of filenames just like HFS does. Just put this in your ~/.inputrc file:
set completion-ignore-case On
Then when you type ls /appli in the shell you'll get ls /Applications/ like you ought to!
Changing all *.oop files to *.ack [permalink]
% ~/bin/rename.pl 's/.oop$/.ack/' *.oop
To tack on an extension to all files, do something like % ~/bin/rename.pl 's/(.*)/$1.jpg/' *
Converting from DOS to unix line terminators [permalink]
% perl -pi -e 's/\r/\n/g' mealcsv.csv
Converting mac newlines to unix newlines in a text file [permalink]
```
tr '\r' '\n' < macfile.txt > unixfile.txt
```

Creating a disk image from the command line [permalink]

% hdiutil create -ov -imagekey zlib-level=9 -fs HFS+ -format UDZO -scrub -srcfolder /Path/To/The/Goodies fooby.dmg

Daemontools controls [permalink]
# svc -d /service/whatever - Bring the server down
# svc -u /service/whatever - Start the server up and leave it in keepalive mode.
# svc -o /service/whatever - Start the server up once. Do not restart it if it stops.
# svc -t /service/whatever - Stop and immediately restart the server.
# svc -k /service/whatever - Sends the server a KILL signal. This is like KILL -9. If svc -t fails to fully kill the process, use this option.
(Thanks to the OpenACS documentation)
Dealing with directories with spaces in them [permalink]
You can either escape the directory name with a space: % cd /Users/bork/Library/Screen\ Savers
or you can surround it with quotes: % cd "/Users/bork/Library/Screen Savers"
Note that tilde expansion (~/Library) is suppressed if you use the quotes.
Deatching another screen session. [permalink]
Sometimes if you have an unclean disconnect, screen can still be attached and you get the smackdown:
```
% screen -r
There is a screen on:
        8395.pts-0.vikki    (Multi, attached)
```
You can boot off that other loser by using:
```
% screen -d 8395.pts-0.vikki
```
and then doing screen-r again
Directory space consumption [permalink]
To see how much disk space a directory consumes, use
% du -sk dir_name
To see how much each of a directory's contents consume, use
% du -sk dir_name/* | sort -n to get a list ordered by size.
Display the info.plist embedded in a compiled file [permalink]
```
% otool -s __TEXT __info_plist ./thebinary | xxd -r
```
Muchos Thankos to Dan Jalkut for this one.
Doing something to a hierarchy of files [permalink]
To do something to all of the html files in a directory (e.g. to grep something)
% cd directory % find . -name "*.h" -exec grep NSDoWhatIMean {} ; -print
Don't forget the dot right after find, and the {} construct. Also make sure there's a space after the {} and ;
Don't show file names with a multi-file grep [permalink]
use the -h flag:
```
% grep -h chicken ~/Documents/ircLogs/FreeNode/2007*/#macsb*
```
Find all the literal strings in an objc source file [permalink]
```
egrep -h -o '@"[^"]*"' MJFirstViewController.m
```
The -h suppresses printing the file name, -o is "Prints only the matching part of the line"

Find files larger than 2 gigs [permalink]

find /Volumes/Vikki/ -type f -size +2G -print

Finding Mac OS X version from the command line [permalink]
% sw_vers

ProductName:    Mac OS X
ProductVersion: 10.3.7
BuildVersion:   7S215

Finding parent process IDs on Linux [permalink]
% ps -eo "%p %P %c"
Finding suid executables [permalink]
To find setuid executables, run
% find . -type f -perm +06000 -print
(or use find / to start looking from the root directory.)
Finding the path corresponding to an open file descriptor [permalink]
With fcntl, all things are possible.
```
char pathbuf[PATH_MAX];

if (fcntl(fd, F_GETPATH, pathbuf) >= 0) {
    printf ("wow: %s\n", pathbuf);
} else {
    printf ("oops %s\n", strerror(errno));
}
```
Finding this feature was a surprise, having lived under the assumption that you can't map from a file descriptor to a path in the file system. Obviously things like sockets and pipes will return an error, but it actually seems to work.
Getting a count from grep [permalink]
To see how many times "fix me" occurs in the Objective-C files in the current directory:
% grep -c "fix me" *.m
Getting curl to download a file [permalink]
% curl -O http://borkware.com/quickies/files/fullscreensaver.tar.gz
Getting grep to return file names of matches rather than the actual match [permalink]
Use the -l (dash ell) flag to grep. If a file contains the expression, the name of the file (rather than the matching line) is displayed. e.g. to open all the files that contain 'rzolf'
% grep -l rzolf *.ham | xargs open
Grepping for flags [permalink]
Sometimes you want to grep for something like "-H", but don't want it to be interpreted as a grep flag. The -e option to the rescue:
```
% grep -e -H filename
```
Interleaving the contents of two files [permalink]
Say file1.txt had the contents
```
hoover
Greeble
NI POP
```
and file2.txt had the contents
```
bork
Bork
BORK
```
Interleave them with
```
% paste -d '\n' file1.txt file2.txt > blarg
```
and get
```
hoover
bork
Greeble
Bork
NI POP
BORK
```
Mounting a disk image from the command line [permalink]
% hdiutil mount diskimage.dmg
Perl breaks in weird and wonderful(?) ways [permalink]
Perl 5.8 uses unicode internally by default, which apparently causes trouble for some big perl packages (like SpamAssassin's makefile getting totally trashed when you create it). Set the LANG environment variable to be en_US to make it happier.
Prevent deep sleep [permalink]
My lappy has 16 gigs of memory (yay me), and when the machine goes in to a deep sleep / hibernation, it effectively powers down. Upon wake, memory has to be read back in from disk, which can take a noticeable amount of time. While that's happening, the screen is blank, and the computer is inert like it has totally died.
Once you've totally finished panicking and starting your recovery plan ("dammit now I'll lose a day's work getting this to a genius and re-cloning my repos to another machine") the memory has finished swapping in and the machine wakes. "Hi! Now that you're completely carbonated, let's get back to work!"
Unsurprisingly, major OS updates seem to reset this value. (Firmware updates may do it too.)
(This may be incomplete - I have a couple of days still to make sure it totally works)
Turn off hibernation with:
```
# sudo pmset -a standby 0
# sudo pmset -a hibernatemode 0
```
Preventing an app from opening stale windows [permalink]
Lion introduced the idea that apps restore all their windows that were last open. Sometimes that's wonderful. For some apps, it's maddening. You can tweak individual apps by setting a default:
```
% defaults write com.bignerdranch.bigshow NSQuitAlwaysKeepsWindows -bool false
```
(thanks to Step Christopher for this one)
Pruning directories with find [permalink]
Sometimes you don't want find to go into a particular subtree. This command finds all swift files from the current directory, except those in experimental:
```
% find . -path ./experimental -prune -o -name "*.swift"
```
Removing files with a leading "-" [permalink]
% rm -- -i-am-a-bad-file

Removing quarantine xattribute from downloaded files. [permalink]
Download a zip file full of graphics, and they all have extra attributes?

% ls -l
total 18192
-rw-r--r--@ 1 markd  staff   13354 Jan  5 18:46 Icon-114.png
-rw-r--r--@ 1 markd  staff   21379 Jan 18 10:56 Icon-152.png
-rw-r--r--@ 1 markd  staff  484470 Jan  5 18:40 Icon-167.png
...

Which are quarantine tags:

% ls -l@
total 18192
-rw-r--r--@ 1 markd  staff   13354 Jan  5 18:46 Icon-114.png
        com.apple.quarantine        61
-rw-r--r--@ 1 markd  staff   21379 Jan 18 10:56 Icon-152.png
        com.apple.quarantine        61
-rw-r--r--@ 1 markd  staff  484470 Jan  5 18:40 Icon-167.png
        com.apple.quarantine        61

Get rid of them with this:

% xattr -d com.apple.quarantine *

Replacing a string in a set of files [permalink]
% perl -pi.bak -e 's/OLDSTRING/NEWSTRING/g' *.html
This will leave "*.bak" files around of the original version, in case the replacement string didn't quite do what you wanted.
Restricting Find to only deal with files (not directories) [permalink]
% find . -type f -name "*.html" -exec grep Spoon {} ; -print
Rsyncing a directory hierarchy from one machine to another [permalink]
I'm moving a bunch of movie files from one machine to another. I used this rsync command to copy it from the remote machine to the local one to the directory /Volumes/Media2
```
% rsync -avz -e ssh --progress 'biggun.local:/Volumes/Video/AppleTV\ Videos/Exercise' /Volumes/Media2
```
I can never remember what the flags mean, so:
- -a : archive mode (recursive, also preserve symlinks, permissions, owner and group)
- -v : verbose
- -z : compress for transport
- -e ssh : use ssh for the transport pipeline
- --progress : see progress info
Running a pipeline in sudo [permalink]
% sudo sh -c "tar cf - * | (cd /mirror4/web; tar xf -)"
Sanity checking an XML file [permalink]
% xmllint -noout kipple.xml Handy for a quick check after hand-editing a file.
Seeing all the flags gcc takes [permalink]
% gcc -v --help

Seeing extended attributes with 'ls' [permalink]
Sometimes you see an @ after the directory permissions in an ls -l:

% ls -l ~/junk/SnapshotRepository.sparseimage
-rw-r--r--@ 1 markd  markd  135270400 Jul 24 20:38 /Users/markd/junk/SnapshotRepository.sparseimage

That means there's some extended attributes. use ls -l@ to see them:

% ls -l@ ~/junk/SnapshotRepository.sparseimage
-rw-r--r--@ 1 markd  markd  135270400 Jul 24 20:38 /Users/markd/junk/SnapshotRepository.sparseimage
	com.apple.diskimages.fsck	       20

Seeing lines that exist in two files [permalink]
Assuming the files are sorted, you can use comm.
Here I'm wondering if any objc files in the current directory also exist in the Classes subdirectory
```
% ls -1 *.[hm] *.xib | sort > oopack
% cd Classes
% ls -1 *.[hm] *.xib | sort > ../oop2
% cd ..
% comm -12 oopack oop2
```
The -12 suppresses lines only in file1, and lines only in file2. I'm more interested in their intersection.
Seeing the console.log without running Console.app [permalink]
In a terminal or an emacs shell:
% tail -f /var/tmp/console.log
Seeing the files in a zip archive [permalink]
% unzip -l snorgle.zip
Seeing the parent process id in ps [permalink]
Use the ppid option with the -o flag:
% ps -axo user,pid,ppid,vsz,tt,state,start,time,command
Silencing Macbook power chime [permalink]
The modern-day flutterby keyboard Macbooks make a soothing (?) chime when plugged in to power, even if the system volume is muted. If you don't want to hear that, you can do
```
% defaults write com.apple.PowerChime ChimeOnNoHardware -bool true
% sudo killall PowerChime
```
Sort grep-c [permalink]
grep -c will give you the number of hits for the given pattern in each file. It'd be nice to sort them, but the count occurs after the filename, e.g. GreebleBork:23
So, trick sort into using the last field (with a colon separator) as the sort key:
```
% grep -c translatesAuto *.xib | sort -t: -n -k2
```
Toggle 'screen's visible bell on and off [permalink]
Assuming ^t is the hotkey, doing ^t ^g will toggle the visible bell on and off.
Turning blobs of json in to something somewhat readable [permalink]
yay prettyprinting!
```
% cat some-unformatted-manifest.json | python -m json.tool
```
(courtesy of Chris Donnelly)
Using "find" with multiple -name clauses [permalink]
% find . $ -name "*.jpg" -or -name "*.gif" $ -exec do_something_with {} ;
Using Image Magick to scale an image proportionally [permalink]
% find . -name "*.jpg" -exec convert -verbose -geometry 150x150 {} {} ;
This converts the files in-place, so have a backup. Even though the geometry is specified as 150x150, that just means make the image no larger than 150 in either direction.
Using ImageMagick to get the dimensions of a file [permalink]
% identify filename.jpg
If you want to be studly, you can use -format to tailor the output
% identify -format "insert into bw_graphics values (generate_primary_key(), '/images/llamas/%f', %w, %h);" *.jpg
generates a sql insert statement for each image.
Using awk to extract the n'th column [permalink]
nm outputs a three-column format. I needed to get the set of defined symbols (in particular those that start with __, in tracking down a C++ linking problem), which is the third column. awk is good for that. Use $N to get the nth column (zero-index). This pipeline did the trick:
```
 nm ./oopack.o | awk '{print $2}' | sort | grep __
```
(Thanks to DougEDoug for the pointer)
Using awk to print fields [permalink]
Here's a way to have awk print all fields (separated by spaces) from each line, except for fields number 1 and 9 (starting with a count of 1):
```
% awk '{ for (f=1; f <= NF; f++) { if (f != 1 && f != 9) printf("%s ", $f);} printf("
");}' < ook.txt
```
Or you can put this at the end of a pipeline instead of directing a text file into the command. (courtesy of DougEDoug)
Using tar to copy across the network [permalink]
% tar cf - ./stuff | ssh theOtherMachine "tar xf -"
cd to directory with a space in its name [permalink]
There's two ways to do it. The first is to backslash-escape the spaces:
% cd /Users/bork/development/cool\ stuff/neat\ things
The other is to use shell completion to put in the backslashes for you. In the above path, if you type
% cd /Users/bork/development/cool[tab], it'll expand "cool stuff" for you.
safely copying directories [permalink]
cp -R is generally unsafe to copy directories with since it will copy through symbolic links rather than just copying the link (newer cp's have an -H flag that will work around this). The classic unix way of doing things like this is a push-pull tar:
% tar cf - dir-name | (cd /destination-dir; tar xf -)
In English: tar out through a pipe into a subshell that's changed directories, feeding that pipe into the tar extraction. If you run both sides of the pipeline as root (via sudo), the file owners and permissions will be preserved (which cp won't do)
tar + xargs for huge numbers of files [permalink]
If you have too many files to add to a tarball, you can run into the shell's command limitation. If you use xargs + tar cf, you'll not get all of your files added to the tarball. Use tar's rf command to append to an archive rather than creating one:
```
% find . -name "*.yuck" -print0 | xargs -0 tar rvf oop.tar
```
-print0 (zero) tells find to output the filenames with trailing NUL characters (rather than newlines), and -0 tells xargs to use a NUL as its input separator, rather than newlines/spaces. The upshot is that you can handle filenames with spaces in them.
Thanks to Ben Cox for the -print0 / -0 suggestion.
using ssh (or CVS) without a password [permalink]
Suppose you have two machines, the local one (your laptop) and the remote one (the honking big server located in Minneapolis that hosts gitdown.com) To ssh (or CVS) from the laptop to the server without needing a password, perform these steps:
1. % ssh-keygen -t rsa
  (don't use a passphrase. Just hit return twice)
2. % scp ~/.ssh/id_rsa.pub gitdown.com:
3. On the remote machine (gitdown)
  % cat ~/id_rsa.pub >> ~/.ssh/authorized_keys

Hacks

I can't move or symlink a directory [permalink]
If you get the error
```
mv: rename build/ to oopack: Is a directory
```
when using mv or ln on the command line, that's a Darwin error. To correct the problem, remove the trailing slash from the directory name.
Printing network traffic [permalink]
```
% sudo tcpdump -Atq -s 0 -i en1
```
-i en1 will display traffic on your airport card. Use en0 (or nothing, for most systems) for built-in ethernettage.
You can add a host line to limit output to a particular host
```
% sudo tcpdump -Atq -s 0 -i en1 host zombo.com
```
(thanks to Dan Jalkut for this one)
Restarting a crashed Dock [permalink]
Ever be using your machine and you can't command-Tab and your Dock is gone? And there's no Dock process to kill? (#itJustWorks). You can restart the Dock with:
```
% launchctl stop com.apple.Dock.agent
% launchctl start com.apple.Dock.agent
```
Unwedging stuck boots [permalink]
If I go longer than a week or so between reboots, it (Mavericks) literally takes forever (I counted) for it to actually boot. Verbose mode shows fsck runs (and actually completes, but it hangs.) If that happens to you regularly, try this:
```
// Boot into single-user mode with command-S on power-on
# /sbin/fsck -fy
# /sbin/mount -uw /
# mv /var/folders /var/folders-dorked
# reboot
```
Nearly instant boot again.

Random

Changing a file's modification date [permalink]
Sometimes you're doing work which uses a file's modification time to decide whether you want to do something (like don't process today's log file, or remove a really old file). You could use a time machine to test this, or use touch. You can specify a specific date or time using the format [[CC]YY]MMDDhhmm[.SS]]. Here's an example with alternating bolds to make it easier to read:
```
% touch -t 200703141536 snork.waffle

% ls -l snork.waffle
-rw-r--r--   1 markd  markd  298 Mar 14 15:36 snork.waffle
```
Creating a RAM Disk [permalink]
% hdid ram://size where size is the size in bytes. the system will round that number up or down as it sees fit.
Getting OS version from the command line [permalink]
```
% sw_vers -productVersion
10.5.1
```

Seeing media types supported by your CD/DVD burner [permalink]
On Panther, you can run /usr/bin/drutil info to get a listing of the kinds of media your drive supports. For example, on the 17" iLamp:

% drutil info
 Vendor   Product           Rev 
 PIONEER  DVD-RW  DVR-104   A227

   Interconnect: ATAPI
   SupportLevel: Apple Shipping
          Cache: 2000k
       CD-Write: -R, -RW, BUFE, Test, IndexPts, ISRC
      DVD-Write: -R, -RW, BUFE, Test
     Strategies: CD-TAO, CD-SAO, DVD-DAO

Setting file system attribute flags [permalink]
chflags in Mac OS X works like chattr over in Linux-land. So to make a file immutable (can't change it, even if you're the superuser), you can sudo chflags uchg file-name(s). To undo it, do sudo chflags nouchg file-name(s)

Tools

Changing the 'screen' hotkey [permalink]
Running emacs inside of screen is made difficult because both want to use C-A for something important. C-O (AFAIK so far) isn't too useful for my use of emacs, so I use that for screen now:
```
screen -e^Oo emacs
```
Rejoining a disconnected screen session [permalink]
```
screen -r
```