Braindump

I’ve previously written about SSH and ssh-agent on Mac OS X where I mentioned a utility named SSHKeychain that helps manage the agent daemon and your passphrases. Well, Mac OS X 10.5 (Leopard) has been released since that post, and things have changed. The long and the short of it is that ssh-agent is handled much better than before, by default. But its usage can also be a bit confusing (at least it was for me). I’ll try and explain how it all works in Leopard, so you can get the most out of it.

Why Use a Passphrase, Reprisal

I read a rebuttal or two to my previous post, where I mentioned that using a passphrase protects your private key, if someone gains access to your account. Their main argument was that if your system gets hacked, then the bad guy has complete access to your machine and you’re screwed anyways. True, if someone gets root on your machine, the game is pretty much over. As root, they can do really nasty things like steal ssh-agent identities and install key loggers to snoop your passphrase.

The problem is that root exploits are not the only way to get hacked. What if a bug in a web browser’s Javascript exposes local filesystem access? A nasty web page could read your private key file (~/.ssh/id_rsa), and post it to a website, for example. Or what if you accidentally left your laptop unattended (and unlocked) for a few minutes at the coffee shop? Someone could grab the private key file and stick it on a USB drive. If the attacker was even half way smart, they’d grab ~/.ssh/known_hosts along with your key, since that contains host names you’ve connected to.

Without a passphrase, your private key is completely usable to the bad guy. Using a passphrase encrypts your private key, so that they would have to crack your passphrase to get access to it. In summary, an empty passprhase on your private key is a bad idea. It’s just asking for trouble.

The Über-server

Okay, back to the topic at hand: ssh-agent on Leopard. One of the benefits of SSHKeychain (or one of the other ssh-agent apps for OS X) is that it starts ssh-agent at login time. It also sets the SSH_AUTH_SOCK environment variable (which points to a Unix domain socket) to be accessible by all apps (usually by modifying ~/.MacOSX/environment.plist). Leopard gives you the equivalent of this, out of the box. Open up a terminal and see:

% echo $SSH_AUTH_SOCK
/tmp/launch-nZRFjA/Listeners
While this environment variable is automatically set for all processes, this is a little deceiving. ssh-agent does not actually get started when you log in. Go ahead and see for you self, by running this command right after you login:

% ps xa | grep ssh-agent | grep -v grep
You should get nothing back. It turns out that ssh-agent gets started on demand, the first time something tries to connect to the socket:

ps xa | grep ssh-agent | grep -v grep ssh-add -l
The agent has no identities.
% ps xa | grep ssh-agent | grep -v grep
10877 ?? S 0:00.02 /usr/bin/ssh-agent -l
The ssh-add -l tried to connect to the socket, and thus caused ssh-agent to launch. Pretty nifty.

But how can the SSH_AUTH_SOCK be valid without ssh-agent? The magic sauce is launchd. launchd was introduced in Mac OS X 10.4 as a replacement for many traditional Unix daemons such as cron, xinetd, and init. Since (x)inetd is known as a super-server, and launchd replaces xinted plus a few other daemons, I like calling launchd an über-server. In this context, launchd creates the Unix domain socket and listens for connections, on behalf of ssh-agent. When something connects, it automatically launches ssh-agent. Since launchd is always running, it can listen for connections right after you login.

For the curious, this is done with the new SecureSocketWithKey plist key for launchd. From launchd.plist(5):

This optional key is a variant of SockPathName. Instead of binding to a known path, a securely generated socket is created and the path is assigned to the environment variable that is inherited by all jobs spawned by launchd.

Rock! The launchd plist file for ssh-agent is at:

/System/Library/LaunchAgents/org.openbsd.ssh-agent.plist
The benefit of this lazy launching is that ssh-agent is only run if you use ssh. If you never use ssh, the agent never gets launched, and you don’t waste any resources running it.

Keychain Integration

But the awesomeness doesn’t stop there. If you have an SSH key and try to connect to a server, you’ll get this nifty dialog box asking for your passphrase:

The first benefit of this dialog box is that it uses a secure text field for your passphrase. This field is not copiable and not snoopable via universal access or low-level keyboard routines. The real benefit, though, is the second checkbox: “Remember password in my keychain.” While it does store the passphrase in your keychain, it actually does more than that. It also adds the identity to your ssh-agent for you:

ssh-add -l

The agent has no identities. ssh dave@example.com
... Type passphrase and check "Remember in my keychain" ...
example.com> exit
Connection to example.com closed.
% ssh-add -l
2048 xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx /Users/dave/.ssh/id_rsa (RSA)
By adding your identity to the agent, you can now log right back into the same machine, without typing any passphrase. However, ssh does not prompt for you passphrase because it gets it from the agent, not your keychain. Remove your identity from the agent, and try again:

ssh-add -D

All identities removed. ssh dave@example.com
You’ll get that same GUI dialog box again. But it stores your passphrase in your keychain, right? Then why is it asking for your passphrase if it’s in your keychain? And, why would it store the passphrase in your keychain if it’s not going to actually use it? I know I was scratching my head at this point. Well, it does use the keychain, just not how you may think.

It turns at that when ssh-agent is started, it automatically adds all identities that have passphrases stored your keychain. So the normal workflow, where identities are not removed from the agent, just works. Since the GUI dialog box added your identity to the agent when it added the passphrase to your keychain, you don’t need enter your passphrase again for the rest of that login session. For all future sessions, the identity is automatically added when ssh-agent starts. And remember from above, that ssh-agent starts on demand, only when needed. Thus, you only enter your passphrase once, and from then on it grabs it from the keychain.

Manually Adding Identities

If you do want to remove your identities, you can manually add all the identities from the keychain with the Apple-specific -k option on ssh-add. From ssh-add(1):

Add identities to the agent using any passphrases stored in your keychain.

Sweet! Let’s try it out:

ssh-add -D

All identities removed. ssh-add -l
The agent has no identities.
ssh-add -k

Added keychain identities. ssh-add -l
2048 xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx /Users/dave/.ssh/id_rsa (RSA)
Beware, the passphrase dialog box is not a GUI version of SSH_ASKPASS, though. Try adding identities without consulting the keychain, and you’ll still get prompted in the terminal:

ssh-add -D

All identities removed. ssh-add
Enter passphrase for /Users/dave/.ssh/id_rsa:
And that covers it. I’m a little too paranoid to use this default behavior, though, as I don’t want my passphrase to be stored in my login keychain. Thus, I’ve written a follow-up article that discusses possible ways to beef up security of your passphrase

Posted by Mario at 01:48 PM | Permalink | Comments (0)

netstat command and shell pipe feature can be used to dig out more information about particular IP address connection. You can find out total established connections, closing connection, SYN and FIN bits and much more. You can also display summary statistics for each protocol using netstat.

This is useful to find out if your server is under attack or not. You can also list abusive IP address using this method.

# netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n

Output:

1 CLOSE_WAIT 1 established)
1 Foreign
3 FIN_WAIT1
3 LAST_ACK
13 ESTABLISHED
17 LISTEN
154 FIN_WAIT2
327 TIME_WAIT

Dig out more information about a specific ip address:

# netstat -nat |grep {IP-address} | awk '{print $6}' | sort | uniq -c | sort -n

2 LAST_ACK 2 LISTEN
4 FIN_WAIT1
14 ESTABLISHED
91 TIME_WAIT
130 FIN_WAIT2

Busy server can give out more information:

# netstat -nat |grep 202.54.1.10 | awk '{print $6}' | sort | uniq -c | sort -n

Output:

15 CLOSE_WAIT 37 LAST_ACK
64 FIN_WAIT_1
65 FIN_WAIT_2
1251 TIME_WAIT
3597 SYN_SENT
5124 ESTABLISHED

Get List Of All Unique IP Address To print list of all unique IP address connected to server, enter:

# netstat -nat | awk '{ print $5}' | cut -d: -f1 | sed -e '/^$/d' | uniq

To print total of all unique IP address, enter:

# netstat -nat | awk '{ print $5}' | cut -d: -f1 | sed -e '/^$/d' | uniq | wc -l

Output:

449

Find Out If Box is Under DoS Attack or Not If you think your Linux box is under attack, print out a list of open connections on your box and sorts them by according to IP address, enter:

# netstat -atun | awk '{print $5}' | cut -d: -f1 | sed -e '/^$/d' |sort | uniq -c | sort -n

Output:

1 10.0.77.52 2 10.1.11.3
4 12.109.42.21
6 12.191.136.3
.....
...
....
13 202.155.209.202
18 208.67.222.222
28 0.0.0.0
233 127.0.0.1

You can simply block all abusive IPs using iptables or just null route them.
Get Live View of TCP Connections

You can use tcptrack command to display the status of TCP connections that it sees on a given network interface. tcptrack monitors their state and displays information such as state, source/destination addresses and bandwidth usage in a sorted, updated list very much like the top command.
Display Summary Statistics for Each Protocol

Simply use netstat -s:

# netstat -s | less # netstat -t -s | less
# netstat -u -s | less
# netstat -w -s | less
# netstat -s

Output: Ip:

88354557 total packets received 0 forwarded
0 incoming packets discarded
88104061 incoming packets delivered
96037391 requests sent out
13 outgoing packets dropped
66 fragments dropped after timeout
295 reassemblies required
106 packets reassembled ok
66 packet reassembles failed
34 fragments failed

Icmp:

18108 ICMP messages received 58 input ICMP message failed.
ICMP input histogram:
destination unreachable: 7173
timeout in transit: 472
redirects: 353
echo requests: 10096
28977 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 18881
echo replies: 10096

Tcp:

1202226 active connections openings 2706802 passive connection openings
7394 failed connection attempts
47018 connection resets received
23 connections established
87975383 segments received
95235730 segments send out
681174 segments retransmited
2044 bad segments received.
80805 resets sent

Udp:

92689 packets received 14611 packets to unknown port received.
0 packet receive errors
96755 packets sent

TcpExt:

48452 invalid SYN cookies received 7357 resets received for embryonic SYN_RECV sockets
43 ICMP packets dropped because they were out-of-window
5 ICMP packets dropped because socket was locked
2672073 TCP sockets finished time wait in fast timer
441 time wait sockets recycled by time stamp
368562 delayed acks sent
430 delayed acks further delayed because of locked socket
Quick ack mode was activated 36127 times
32318597 packets directly queued to recvmsg prequeue.
741479256 packets directly received from backlog
1502338990 packets directly received from prequeue
18343750 packets header predicted
10220683 packets header predicted and directly queued to user
17516622 acknowledgments not containing data received
36549771 predicted acknowledgments
102672 times recovered from packet loss due to fast retransmit
Detected reordering 1596 times using reno fast retransmit
Detected reordering 1 times using time stamp
8 congestion windows fully recovered
32 congestion windows partially recovered using Hoe heuristic
19 congestion windows recovered after partial ack
0 TCP data loss events
39951 timeouts after reno fast retransmit
29653 timeouts in loss state
197005 fast retransmits
186937 retransmits in slow start
131433 other TCP timeouts
TCPRenoRecoveryFail: 20217
147 times receiver scheduled too late for direct processing
29010 connections reset due to unexpected data
365 connections reset due to early user close
6979 connections aborted due to timeout

Display Interface Table You can easily display dropped and total transmitted packets with netstat for eth0:

# netstat --interfaces=eth0

Output:

Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0 1500 0 2040929 0 0 0 3850539 0 0 0 BMRU

Posted by Mario at 11:59 AM | Permalink | Comments (0)

LiSt Open Files (LSOF) is a Linux utility that allows you to view current network connections and the files associated with them.

LSOF provides verbose output and is useful in tracking down all sorts of information. For example, it allows you to see what program is operating on an open port, which daemons have established connections, and what ports are open on your server.

While similar in many ways to utilities like netstat and fuser, LSOF has many unique options that let you find specific information on ports, users, processes, and files.

Getting and installing LSOF

On many Linux systems, LSOF will be installed by default. Try running lsof -v to see whether the program exists on your system. If it doesn't, you will need to download one of the many packages available and install it yourself. Source and binary versions are available here, while distribution-specific packages are available from locations such as Rpmfind.net and are included in databases for apt-get, up2date, and urpmi.

If you are installing from source, download the tarball and perform these commands:

tar xpfz lsof_4.64.tar.gz
cd lsof_4.64
tar xpf lsof_4.64_src.tar
cd lsof_4.64_src
./Configure linux make

During the configuration, you have the option of running an inventory to verify that all the files necessary for compilation are present. You will be asked if you want to customise the installation. This shouldn't be necessary on most Linux platforms, so answering no is fine. If you encounter any problems running LSOF later, you may want to come back and try some customisation. Customisation may also be needed if you upgrade or modify your kernel.

After running make, the executable lsof should appear in the directory. There is no default make install rule, so you can create your own or just copy lsof to your directory of choice. Keep in mind that you may need to modify its permissions to be setuid-root if you want regular users to be able to see all open files. Not that this is necessarily recommended, but it's good to remember, since the output generated when LSOF is run as root will be different from the output generated when it's run by a normal user account.

Looking at network connections

As its name indicates, LSOF deals with open files on a Linux system. An open file can be a regular file, a directory, a library, a stream, or a network socket. You can take advantage of various LSOF options, depending on what you're looking for.

Running lsof by itself will output all open files corresponding to every active process on the box. This can be quite lengthy, so it's best to know what you are looking for in advance. You can get a quick rundown of the options with lsof -h, and the man page goes into much further detail. Let's take a look at some common switches and what they will show us.

lsof -i

The lsof -i command lists all open files associated with Internet connections. It is similar in format to netstat -a -p and will look something like Listing A.

By default, LSOF lists detailed information about each connection. In Listing A, we see the command or program involved, the process ID (PID), the user running the command, the file descriptor (FD), the type of connection, the device number, the Internet protocol, and the name of the file or Internet address. The -i option can be useful when you're attempting to secure your Linux box. You can quickly determine what ports are open and listening for incoming connections. LSOF will also associate them with a program name. Thus, you can quickly identify unnecessary security risks and shut them down.

Ports that are awaiting connections have the keyword LISTEN appended to them. These are ports that are open and accepting connections. Note that LSOF will not distinguish between ports that are completely open and ones that have filters applied. The keyword ESTABLISHED indicates that a connection on the given port has been made. In Listing A, there is an SSH session from labrat.remote.net to test.com. You can see multiple processes associated with the sshd daemon. The main daemon, PID 597, handles incoming requests and forks itself as needed. PID 8545 was spawned by sshd and is responsible for the 8547 process. The only noticeable difference between 8545 and 8547, besides the PID, is in the user field. Notice that lhutz is the user who has remotely logged in to this box. This is useful information that goes beyond merely presenting network connections.

You can narrow your search by specifying a particular port, service, or host name using techniques such as:

lsof -i :587
lsof -i :smtp
lsof -i @labrat.remote.net

LSOF will then output all matching connections. The above examples will list connections listening or established on port 587, list connections associated with the well-known service SMTP, and list connections coming from or going to the host labrat.remote.net, respectively. These techniques are handy if you know what you are looking for in advance. You can watch and see whether inbound SMTP connections are taking too long, possibly causing timeouts. You can verify that the service is in fact running and what port it is listening on. And you can see if anyone from a certain device is connected to your system, whether it is via SSH, Telnet, FTP, or just about any other way possible.

lsof -p 409

LSOF will also accept a PID and output all open files it is using. In this particular instance, we performed an lsof -I to determine what PID number NameD (BIND DNS service) was operating under. Once we discovered it was 409, we issued the command lsof -p 409. The output is shown in Listing B.

You'll notice the different FDs, or file descriptors, right away. The cwd variable represents the current working directory of the process; txt defines the program text, which is the executable itself; mem is a file held in memory, in this case a library; the 4 and 21 represent files in use by this particular process; and the u designator defines them as having both read and write access. These all help you determine whether something physically exists on the system, is being used by the process, or is being held in memory.

lsof +d

The command lsof +d /var/log/apache/ is similar to fuser. It basically associates open files with their processes. In this case, we are looking at all regular files in the /var/log/apache/ directory. The output would look something like Listing C. In this example, Apache is keeping track of two sets of log files, an access and an error log for two domains. As you can see, there are some differences between regular files and Internet connections. For one thing, the TYPE is now REG, indicating a regular file. Also, a SIZE variable is present, which indicates the actual size in bits the file takes up. Notice too that the DEV variable indicates they all use the same device, in this example, a single hard drive. The +d flag that was issued with LSOF tells the command not to leave the top-level directory, while +D would perform a recursive check on all subdirectories.

lsof -F <...>

The -F switch provides an excellent way to format LSOF output. This built-in feature allows you to pipe information directly into external programs, such as a Perl script, a C program, or even a monitoring program like MRTG. You do this by specifying which fields you would like printed. For example, lsof -F pcfn would print the process ID, the command name, the file descriptor, and the filename. Many options are available, and this can save you time in working with the raw data yourself.

We've covered just a few of the options LSOF provides, but the man page covers the full spectrum of LSOF capabilities. Some LSOF commands may be processor intensive, due to the sheer number of processes on a system, so be as selective as possible when running commands.

Summary

LSOF is an excellent utility for managing and tracking network connections on your Linux system. Although a number of utilities can perform similar functions, none is quite as robust as LSOF. With LSOF, you can list open ports, identify connections currently being made to your system, and determine what resources a process is using. Not only that, but you can also determine what processes a particular user has and find detailed information about file and directory usage.

Posted by Mario at 11:56 AM | Permalink | Comments (0)

The package included with Plesk 8.3 (on linux) does not include RDoc. If you try to update gem using the command: gem update --system... the installation will exit with an error. You need to install ruby-rdoc (using yum install ruby-rdoc) and its dependencies before re-running gem update --system. Then it will update Rails and all the other gem to their latest versions.

Posted by Mario at 05:16 PM | Permalink | Comments (0)

To make the app show up at the document root, we have to create a vhost.conf file in your domain's conf directory, and then use the websrvmng application to regenerate your domain's configuration. Change directory to the domain's conf directory: Code: cd /var/www/vhosts/{domainname.com}/conf/ Create and begin editing a vhost.com file: Code: touch vhost.conf nano vhost.conf Add the following line, replacing data in brackets: Code: DocumentRoot /var/www/vhosts/{domainname.com}/httpdocs/{railsapp}/public Reconfigure the virtualhost using websrvmng: Code: /usr/local/psa/admin/sbin/websrvmng -u --vhost-name={domainname.com} Verify that the following line is in /var/www/vhosts/{domainname.com}/conf/httpd.include (usually at the bottom): Code: Include /var/www/vhosts/{domainname.com}/conf/vhost.conf Restart apache: Code: apache2ctl graceful Visit your site: Code: http://www.{domainname.com}

Posted by Mario at 05:05 PM | Permalink | Comments (0)

Ruby on Rails - WORKING This is how I got Ruby on Rails (RoR) working on my Plesk machine. I didn't find any major problems with Plesk's implementation, although it took me about 6 hours of trial-and-error to get it working right. SWSoft - you could seriously use some documentation for each Operating System you support.

First thing's first: In your Plesk control panel, make sure that you have your server updated with Ruby On Rails is installed. To do so, go to Server > Updater and enable "Ruby on Rails Support."

After this, go to your domain, and enable FastCGI, which will allow your site to use RoR.

Change to your httpdocs directory:

Code:

cd /var/www/vhosts/{domainname.com}/httpdocs

Create a basic rails application. To do so, use this command:

Code:

rails {railsapp}

Remember to replace {railsapp} with the name of your application.

Change the file ownership to the domains user/group:

Code:

chown {user}:{group} {railsapp}

Change the permissions on the sessions directory:

Code:

chmod 777 {railsapp}/tmp/sessions

Note: Someone let me know if this is a bad idea ... I don't think it is, but I'm not sure. My system had problems writing to the sessions folder unless I chmod'd 777.

Change directory to your new application and edit your .htaccess file in the public/ directory:

Code:

nano {railsapp}/public/.htaccess

On line 2, change:
Code:

AddHandler fastcgi-script .fcgi

To:
Code:

AddHandler fcgid-script .fcgi

On line 32, change:
Code:

RewriteRule ^(.*)$ dispatch.cgi [QSA,L]

To:
Code:

RewriteRule ^(.*)$ dispatch.fcgi [QSA,L]

Test your installation. Go to http://www.{domainname.com}/{railsapp}/public/

You should see the standard "Welcome Aboard" Rails index page.

Posted by Mario at 04:58 PM | Permalink | Comments (0)