Friday, August 21, 2009

crontab: unexpected end of line

Whenever you modify your crontabe file, the error “Your “crontab” on server unexpected end of line. This entry has been ignored” is sent to the users email. This happens if there is a blank line in your crontab file.

For example, in the following crontab file there is a blank line between the last two cron jobs.

root@solaris# crontab -l
# The root crontab should be used to perform accounting data collection.
# The rtc command is run to adjust the real time clock if and when
# daylight savings time changes.
10 1 * * 0,4 /etc/cron.d/logchecker
10 2 * * 0 /usr/lib/newsyslog
15 3 * * 0 /usr/lib/fs/nfs/nfsfind

30 4 * * * /usr/local/bin/disk_check,sh

The solution for this problem is to edit the crontab file and look for the blank line and delete the line. In the above, after editing the crontab, it should look like the following:

root@solaris# crontab -l
# The root crontab should be used to perform accounting data collection.
# The rtc command is run to adjust the real time clock if and when
# daylight savings time changes.
10 1 * * 0,4 /etc/cron.d/logchecker
10 2 * * 0 /usr/lib/newsyslog
15 3 * * 0 /usr/lib/fs/nfs/nfsfind
30 4 * * * /usr/local/bin/disk_check,sh

Read the rest of this entry...

Bookmark and Share My Zimbio

Thursday, August 20, 2009

Find WWN (world wide name) in Solaris

First off, what is WWN anyway?

World Wide Name (WWN) are unique 8 byte (64-bit) identifiers in SCSI or fibre channel similar to that of MAC Addresses on a Network Interface Card (NIC).

Talking about the WWN names, there are also

World Wide port Name (WWpN), a WWN assigned to a port on a Fabric which is what you would be looking for most of the time.

World Wide node Name (WWnN), a WWN assigned to a node/device on a Fibre Channel fabric.

To find the WWN numbers of your HBA card in Sun Solaris, you can use one the following procedures

Using fcinfo (Solaris 10 only)

This is probably the easiest way to find the WWN numbers on your HBA card. Here you can see the HBA Port WWN (WWpN) and the Node WWN (WWnN) of the two ports on the installed Qlogic HAB card.

This is also useful in finding the Model number, Firmwar version FCode, supported and current speeds and the port status of the HBA card/port.

root@ sunserver:/root# fcinfo hba-port | grep WWN
HBA Port WWN: 2100001b32xxxxxx
Node WWN: 2000001b32xxxxxx
HBA Port WWN: 2101001b32yyyyyy
Node WWN: 2001001b32yyyyyy

For detailed info including Make & model number, Firmware, Fcode and current status and supported/current speeds then

root@ sunserver:/root# fcinfo hba-port
HBA Port WWN: 2100001b32xxxxxx
OS Device Name: /dev/cfg/c2
Manufacturer: QLogic Corp.
Model: 375-3356-02
Firmware Version: 4.04.01
FCode/BIOS Version: BIOS: 1.24; fcode: 1.24; EFI: 1.8;
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 4Gb
Node WWN: 2000001b32xxxxxx
HBA Port WWN: 2101001b32yyyyyy
OS Device Name: /dev/cfg/c3
Manufacturer: QLogic Corp.
Model: 375-3356-02
Firmware Version: 4.04.01
FCode/BIOS Version: BIOS: 1.24; fcode: 1.24; EFI: 1.8;
Type: unknown
State: offline
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: not established
Node WWN: 2001001b32yyyyyy

Using scli

root@ sunserver:/root# scli -i | egrep �Node Name|Port Name�
Node Name : 20-00-00-1B-32-XX-XX-XX
Port Name : 21-00-00-1B-32-XX-XX-XX
Node Name : 20-01-00-1B-32-YY-YY-YY
Port Name : 21-01-00-1B-32-YY-YY-YY

For more detailed info on the HBA Cards run as follows: Similar to fcinfo but also provides Model Name and serial number.

root@ sunserver:/root# scli -i
Host Name : sunserver
HBA Model : QLE2462
HBA Alias :
Port : 1
Port Alias :
Node Name : 20-00-00-1B-32-XX-XX-XX
Port Name : 21-00-00-1B-32-XX-XX-XX
Port ID : 11-22-33
Serial Number : AAAAAAA-bbbbbbbbbb
Driver Version : qlc-20080514-2.28
FCode Version : 1.24
Firmware Version : 4.04.01
HBA Instance : 2
OS Instance : 2
HBA ID : 2-QLE2462
OptionROM BIOS Version : 1.24
OptionROM FCode Version : 1.24
OptionROM EFI Version : 1.08
OptionROM Firmware Version : 4.00.26
Actual Connection Mode : Point to Point
Actual Data Rate : 2 Gbps
PortType (Topology) : NPort
Total Number of Devices : 2
HBA Status : Online
Host Name : sunserver
HBA Model : QLE2462
HBA Alias :
Port : 2
Port Alias :
Node Name : 20-01-00-1B-32-YY-YY-YY
Port Name : 21-01-00-1B-32-YY-YY-YY
Port ID : 00-00-00
Serial Number : AAAAAAA-bbbbbbbbbb
Driver Version : qlc-20080514-2.28
FCode Version : 1.24
Firmware Version : 4.04.01
HBA Instance : 3
OS Instance : 3
HBA ID : 3-QLE2462
OptionROM BIOS Version : 1.24
OptionROM FCode Version : 1.24
OptionROM EFI Version : 1.08
OptionROM Firmware Version : 4.00.26
Actual Connection Mode : Unknown
Actual Data Rate : Unknown
PortType (Topology) : Unidentified
Total Number of Devices : 0
HBA Status : Loop down

Using prtconf

root@ sunserver:/root# prtconf -vp | grep -i wwn
port-wwn: 2100001b.32xxxxxx
node-wwn: 2000001b.32xxxxxx
port-wwn: 2101001b.32yyyyyy
node-wwn: 2001001b.32yyyyyy

Using prtpicl

root@ sunserver:/root# prtpicl -v | grep wwn
:node-wwn 20 00 00 1b 32 xx xx xx
:port-wwn 21 00 00 1b 32 xx xx xx
:node-wwn 20 01 00 1b 32 yy yy yy
:port-wwn 21 01 00 1b 32 yy yy yy

Using luxadm

Run the following command to obtain the physical path to the HBA Ports

root@ sunserver:/root$ luxadm -e port
/devices/pci@400/pci@0/pci@9/SUNW,qlc@0/fp@0,0:devctl CONNECTED
/devices/pci@400/pci@0/pci@9/SUNW,qlc@0,1/fp@0,0:devctl NOT CONNECTED

With the physical path obtained from the above command, we can trace the WWN numbers as follows. here I use the physical path to the one that is connected:

root@ sunserver:/root$ luxadm -e dump_map /devices/pci@400/pci@0/pci@9/SUNW,qlc@0/fp@0,0:devctl
Pos Port_ID Hard_Addr Port WWN Node WWN Type
0 123456 0 1111111111111111 2222222222222222 0�0 (Disk device)
1 789123 0 1111111111111111 2222222222222222 0�0 (Disk device)
2 453789 0 2100001b32xxxxxx 2000001b32xxxxxx 0�1f (Unknown Type,Host Bus Adapter)

Read the rest of this entry...

Bookmark and Share My Zimbio

Wednesday, August 19, 2009

ZFS boot/root - backup and restore

Today we will look at backup and restore of a ZFS root pool using native ZFS tools.

The command “ufsdump” does not work on a ZFS file system. This should not surprise anybody.

“flashbackup” will supposedly work if you have all the right patches. I recommend avoiding it until Solaris 10 update 8 is released.

The command “zfs send” is used to create backup images.
The command “zfs receive” is used to restore from backup images.

Some of you may ask: “Do we need backup/restore if we have snapshots?” The answer is yes… if you are performing dangerous maintenance on a critical system, it may be wise to have more than one rollback plan.

Additionally, backup/restore may be useful when migrating an OS between servers, or shinking the size of a root pool.

And some of you will ask: “Can’t we just restore the OS from Netbackup?” The answer is yes… but it will take much longer than using native ZFS tools.

I am not advocating that “zfs send” and “zfs receive” be a replacement for regularly scheduled Netbackup backups; instead I recommend that these commands be used when there is a high probability that a restore of a ZFS root pool will be required.

Backup a ZFS root pool

ZFS backups are done from snapshots. This ensures “single point in time” consistency.

It is simple to recursively create snapshots of all datasets in the root pool with a single command:

# SNAPNAME=`date +%Y%m%d`

# zfs snapshot -r rpool@$SNAPNAME

Backups can be saved to local disk, remote disk, tape, DVD, punch cards, etc. I recommend using an NFS server.

It is possible to backup all the snapshots with a single command. But I don’t recommend this unless you wish to have the contents of the swap and dump devices included in the backup. Backups of swap and dump can be avoided by splitting the backup into four separate commands.

Start with a non-recursive backup of the top level dataset (rpool); then perform recursive “replication stream” backups of rpool/ROOT, rpool/home, and rpool/tools.

You might wish to embed the hostname and date in your filenames, but for this example I will use simple filenames.

# zfs send -v rpool@$SNAPNAME > /net/NFS_SERVER/BACKUP_DIR/rpool.zfs_snap

# zfs send -Rv rpool/ROOT@$SNAPNAME > /net/NFS_SERVER/BACKUP_DIR/root.zfs_snap

# zfs send -Rv rpool/tools@$SNAPNAME > /net/NFS_SERVER/BACKUP_DIR/tools.zfs_snap

# zfs send -Rv rpool/home@$SNAPNAME > /net/NFS_SERVER/BACKUP_DIR/home.zfs_snap

Now list the contents of the backup directory… we should see four backup files.


total 4957206

-rw-r--r-- 1 root root 2369414848 Aug 12 15:45 root.zfs_snap

-rw-r--r-- 1 root root 690996 Aug 12 15:46 home.zfs_snap

-rw-r--r-- 1 root root 92960 Aug 12 15:43 rpool.zfs_snap

-rw-r--r-- 1 root root 166600992 Aug 12 15:46 tools.zfs_snap

The OS backup is now complete.

You may wish to view and record the size of the swap and dump datasets for future use.

# zfs get volsize rpool/dump rpool/swap


rpool/dump volsize 1G -

rpool/swap volsize 8G -

Also, if there are multiple boot environments it is a good idea to record which one is currently in use.

# df -k /

Filesystem 1024-blocks Used Available Capacity Mounted on

rpool/ROOT/blue 70189056 1476722 58790762 3% /


Pick a server and boot it from cdrom or network

Pick a server to restore to.

Depending on your requirements, you can restore to the original server or to a different server.

If you restore to a different server, the CPU architecture must match that of the original server.

e.g. don’t try to restore a backup from a sun4v server to a sun4u server.

In my testing I was able to recover from a 480R to a V210 without any problems.

Boot the server from the network or cdrom using a recent release of Solaris.

ok> boot net

Wait for the server to boot.

Follow the prompts to exit the installation program (usually Esc-2, Esc2, Esc-2, Esc-5, Esc2).


Prepare the disks

Pick a pair of boot disks. The disks do not need to be the same size as the disks on the original system.

If you are using the original disks on the original system you can skip this entire section and jump to the restore.

If you have an x86 system, you need to first make sure there is a valid fdisk partition with type “Solaris 2”; it needs to be marked as “active”.

Here is a sample fdisk layout:

Partition Status Type Start End Length %

========= ====== ============ ===== === ====== ===

1 Active Solaris2 1 8923 8923 100

Regardless of whether you are using a sparc or x86 system, the disks must be labelled with SMI labels and a valid VTOC.

Do not use EFI labels!

Keep in mind that:

When you create a VTOC on a sparc system it applies to the entire disk.

When you create a VTOC on an x86 system it applies to the first fdisk partition of type “Solaris 2”. i.e. in the x86 world a disk is not a disk.

Only one slice is required on each disk… generally it is recommended to use slice #0.

Usually it will make sense to use the entire disk, but if you don’t wish to have your root pool occupying the entire disk then size slice #0 to fit your needs. If you have an x86 system, you should avoid cylinder 0.

Here is a sample VTOC for a sparc server:

Part Tag Flag Cylinders Size Blocks

0 root wm 0 - 14086 68.35GB (14087/0/0) 143349312

1 unassigned wu 0 0 (0/0/0) 0

2 backup wu 0 - 14086 68.35GB (14087/0/0) 143349312

3 unassigned wm 0 0 (0/0/0) 0

4 unassigned wm 0 0 (0/0/0) 0

5 unassigned wm 0 0 (0/0/0) 0

6 unassigned wm 0 0 (0/0/0) 0

7 unassigned wm 0 0 (0/0/0) 0

Here is a sample VTOC for an x86 server.

Part Tag Flag Cylinders Size Blocks

0 root wm 1 - 8920 68.33GB (8920/0/0) 143299800

1 unassigned wm 0 0 (0/0/0) 0

2 backup wu 0 - 8920 68.34GB (8921/0/0) 143315865

3 unassigned wm 0 0 (0/0/0) 0

4 unassigned wm 0 0 (0/0/0) 0

5 unassigned wm 0 0 (0/0/0) 0

6 unassigned wm 0 0 (0/0/0) 0

7 unassigned wm 0 0 (0/0/0) 0

8 boot wu 0 - 0 7.84MB (1/0/0) 16065

9 unassigned wm 0 0 (0/0/0) 0

Install the bootblocks on both disks

For sparc systems run:

# installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c0t2d0s0

# installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c0t3d0s0


For x86 systems run:

# installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c0t2d0s0

# installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c0t3d0s0

Configure the server to use one of the new disks as the boot disk.

For sparc systems this can be done using luxadm(1M).

# luxadm set_boot_dev /dev/dsk/c0t2d0s0


Create a new pool and perform the restore

Mount the file system where the backup images are located:


# mount -o ro NFS_SERVER:BACKUP_DIR /mnt

# ls -1 /mnt





Create a new pool using the syntax shown here. Change only the disk names.

And make sure to include the slices in the names. e.g. c0t2d0s0 and not c0t2d0.

If you only have one disk, the “mirror” keyword must be dropped.

But don’t drop the “mirror” keyword if you specify more than one disk or you will hit problems later on.

# zpool create -f -o failmode=continue -R /a -m legacy -o cachefile=/etc/zfs/zpool.cache rpool mirror c0t2d0s0 c0t3d0s0

Start the restore; the datasets will be automatically created.

# zfs receive -Fd rpool < /mnt/rpool.zfs_snap

# zfs receive -Fd rpool < /mnt/root.zfs_snap

# zfs receive -Fd rpool < /mnt/home.zfs_snap

# zfs receive -Fd rpool < /mnt/tools.zfs_snap

Optionally you may wish to view the recovered datasets

# zfs list -t filesystem -o name,mountpoint,mounted


rpool legacy no

rpool/ROOT legacy no

rpool/ROOT/blue /a yes

rpool/ROOT/blue/var /a/var yes

rpool/home /a/home yes

rpool/tools none no

rpool/tools/marimba /a/opt/Marimba yes

rpool/tools/openv /a/usr/openv yes

rpool/tools/bmc /opt/bmc yes

Create datasets for swap and dump.

# zfs create -V 8G -b 8k rpool/swap

# zfs create -V 1G rpool/dump

Set the default boot envionment

# zpool set bootfs=rpool/ROOT/blue rpool

Note: if you created the pool with multiple disk devices and forgot to specify the “mirror” keyword, this command will fail.

Note: if you have EFI labels on the boot disks this command will fail.

Note: if you followed the instructions above, then everything should work perfectly!

It is not necessary to rebuild the device entries prior to rebooting, even if you are migrating to completely different hardware!

In fact, the system will boot fine even if /dev/dsk and /dev/rdsk are empty directories.

If the restore is part of a migration, you may safely edit /a/etc/nodename, /a/etc/hostname*, /a/etc/inet/hosts, etc. Otherwise move on to the next step.


Cross Fingers, reboot, and pray

# touch /a/reconfigure

# reboot

Wait for the server to reboot. The operating system should come up looking like it did before.

If you have recovered to completely different hardware, you may need to modify the network interface files (/etc/hostname.*) to match the network devices. If necessary, all the network devices can be temporarily plumbed by running “ifconfig –a plumb”

Finally, you may notice that the snapshots still exist. They can be recursively removed at your convience.


# zfs list -t snapshot


rpool@20090813 17K - 35.5K -

rpool/ROOT@20090813 0 - 18K -

rpool/ROOT/blue@20090813 5.07M - 1.41G -

rpool/ROOT/blue/var@20090813 634K - 312M -

rpool/home@20090813 0 - 447K -

rpool/tools@20090813 0 - 18K -

rpool/tools/marimba@20090813 353K - 125M -

rpool/tools/openv@20090813 0 - 27.1M -

# zfs destroy -r rpool@20090813


Reward yourself for pulling off an OS recovery in 15 minutes flat.

i.e go have a an ice cream.

Read the rest of this entry...

Bookmark and Share My Zimbio

Tuesday, August 18, 2009

ZuiTube - Youtube for kids

Yes, you read it right. There's a new video site catered for our little ones and maybe a bunch of us :). First there was Kidzui, the company behind the child-safe web browser of the same name anad now they did it again with its online video experience with ZuiTube, a kid-friendly video destination site.

The main tag lines for ZuiTube are "Play, Laugh, Learn and Share". They boasts of having the largest video collection for kids anywhere, Videos approved by parents and teachers, Channels created by kids and editors,TV mode that plays all videos,search with suggestions and KidRank.

ZuiTube, like Kidzui is free to use. The company makes its money by selling premium memberships that give parents deeper controls and kids more personalization tools and virtual goods.

Other players in the independent kid vid space include Totlol, which had to put up a registration wall in order to stay in business, and Kideo, which is a browser-based video player that pulls up random clips on a continuous cycle.

Read the rest of this entry...

Bookmark and Share My Zimbio

Thursday, August 13, 2009

OpenSSH Server Best Security Practices

OpenSSH is a FREE version of the SSH connectivity tools that technical users of the Internet rely on. Users of telnet, rlogin, and ftp may not realize that their password is transmitted across the Internet unencrypted, but it is. OpenSSH encrypts all traffic (including passwords) to effectively eliminate eavesdropping, connection hijacking, and other attacks. Additionally, OpenSSH provides secure tunneling capabilities and several authentication methods, and supports all SSH protocol versions.

Currently, almost all communications in computer networks are done without encryption. As a consequence, anyone who has access to any machine connected to the network can listen in on any communication. This is being done by hackers, curious administrators, employers, criminals, industrial spies, and governments. Some networks leak off enough electromagnetic radiation that data may be captured even from a distance.

When you log in, your password goes in the network in plain text. Thus, any listener can then use your account to do any evil he likes. Many incidents have been encountered worldwide where crackers have started programs on workstations without the owner's knowledge just to listen to the network and collect passwords. Programs for doing this are available on the Internet, or can be built by a competent programmer in a few hours.

Businesses have trade secrets, patent applications in preparation, pricing information, subcontractor information, client data, personnel data, financial information, etc. Currently, anyone with access to the network (any machine on the network) can listen to anything that goes in the network, without any regard to normal access restrictions.

Many companies are not aware that information can so easily be recovered from the network. They trust that their data is safe since nobody is supposed to know that there is sensitive information in the network, or because so much other data is transferred in the network. This is not a safe policy.

Before implementing these here are the config files and locations:

Default Config Files and SSH Port

* /etc/ssh/sshd_config - OpenSSH server configuration file.
* /etc/ssh/ssh_config - OpenSSH client configuration file.
* ~/.ssh/ - Users ssh configuration directory.
* ~/.ssh/authorized_keys or ~/.ssh/authorized_keys - Lists the public keys (RSA or DSA) that can be used to log into the user’s account
* /etc/nologin - If this file exists, sshd refuses to let anyone except root log in.
* /etc/hosts.allow and /etc/hosts.deny : Access controls lists that should be enforced by tcp-wrappers are defined here.
* SSH default port : TCP 22

Here are Some best OpenSSH Server security practice that you can implement:

#1: Disable OpenSSH Server

Workstations and laptop can work without OpenSSH server. If you need not to provide the remote login and file transfer capabilities of SSH, disable and remove the SSHD server. CentOS / RHEL / Fedora Linux user can disable and remove openssh-server with yum command:

# chkconfig sshd off
# yum erase openssh-server

Debian / Ubuntu Linux user can disable and remove the same with apt-get command:

# apt-get remove openssh-server

You may need to update your iptables script to remove ssh exception rule. Under CentOS / RHEL / Fedora edit the files /etc/sysconfig/iptables and /etc/sysconfig/ip6tables.

Once done restart iptables service:

# service iptables restart
# service ip6tables restart

#2: Only Use SSH Protocol 2

SSH protocol version 1 (SSH-1) has man-in-the-middle attacks problems and security vulnerabilities. SSH-1 is obsolete and should be avoided at all cost. Open sshd_config file and make sure the following line exists:

Protocol 2

#3: Limit Users' SSH Access

By default all systems user can login via SSH using their password or public key. Sometime you create UNIX / Linux user account for ftp or email purpose. However, those user can login to system using ssh. They will have full access to system tools including compilers and scripting languages such as Perl, Python which can open network ports and do many other fancy things. One of my client has really outdated php script and an attacker was able to create a new account on the system via a php script. However, attacker failed to get into box via ssh because it wasn't in AllowUsers.

Only allow root, vivek and jerry user to use the system via SSH, add the following to sshd_config:

AllowUsers root vivek jerry

Alternatively, you can allow all users to login via SSH but deny only a few users, with the following line:

DenyUsers saroj anjali foo

You can also configure Linux PAM allows or deny login via the sshd server. You can allow list of group name to access or deny access to the ssh.

#4: Configure Idle Log Out Timeout Interval

User can login to server via ssh and you can set an idel timeout interval to avoid unattended ssh session. Open sshd_config and make sure following values are configured:

ClientAliveInterval 300
ClientAliveCountMax 0

You are setting an idle timeout interval in seconds (300 secs = 5 minutes). After this interval has passed, the idle user will be automatically kicked out (read as logged out).See how to automatically log BASH / TCSH / SSH users out after a period of inactivity for more details.

#5: Disable .rhosts Files

Don't read the user's ~/.rhosts and ~/.shosts files. Update sshd_config with the following settings:

IgnoreRhosts yes

SSH can emulate the behavior of the obsolete rsh command, just disable insecure access via RSH.

#6: Disable Host-Based Authentication

To disable host-based authentication, update sshd_config with the following option:

HostbasedAuthentication no

#7: Disable root Login via SSH

There is no need to login as root via ssh over a network. Normal users can use su or sudo (recommended) to gain root level access. This also make sure you get full auditing information about who ran privileged commands on the system via sudo. To disable root login via SSH, update sshd_config with the following line:

PermitRootLogin no

#8: Enable a Warning Banner

Set a warning banner by updating sshd_config with the following line:

Banner /etc/issue

#8: Firewall SSH Port # 22

You need to firewall ssh port # 22 by updating iptables or pf firewall configurations. Usually, OpenSSH server must only accept connections from your LAN or other remote WAN sites only.

Netfilter (Iptables) Configuration

Update /etc/sysconfig/iptables (Redhat and friends specific file) to accept connection only from and, enter:

-A RH-Firewall-1-INPUT -s -m state --state NEW -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -s -m state --state NEW -p tcp --dport 22 -j ACCEPT

If you've dual stacked sshd with IPv6, edit /etc/sysconfig/ip6tables (Redhat and friends specific file), enter:

-A RH-Firewall-1-INPUT -s ipv6network::/ipv6mask -m tcp -p tcp --dport 22 -j ACCEPT

Replace ipv6network::/ipv6mask with actual IPv6 ranges.

*BSD PF Firewall Configuration

If you are using PF firewall update /etc/pf.conf as follows:

pass in on $ext_if inet proto tcp from {,} to $ssh_server_ip port ssh flags S/SA synproxy state

#9: Change SSH Port and Limit IP Binding

By default SSH listen to all available interfaces and IP address on the system. Limit ssh port binding and change ssh port (by default brute forcing scripts only try to connects to port # 22). To bind to and IPs and to port 300, add or correct the following line:

Port 300

A better approach to use proactive approaches scripts such as fail2ban or denyhosts (see below).

#10: Use Strong SSH Passwords and Passphrase

It cannot be stressed enough how important it is to use strong user passwords and passphrase for your keys. Brute force attack works because you use dictionary based passwords. You can force users to avoid passwords against a dictionary attack and use john the ripper tool to find out existing weak passwords. Here is a sample random password generator (put in your ~/.bashrc):

genpasswd() {
local l=$1
[ "$l" == "" ] && l=20
tr -dc A-Za-z0-9_ < /dev/urandom | head -c ${l} | xargs

Run it:

genpasswd 16



#11: Use Public Key Based Authentication

Use public/private key pair with password protection for the private key. See how to use RSA and DSA key based authentication. Never ever use passphrase free key (passphrase key less) login.

#12: Use Keychain Based Authentication

keychain is a special bash script designed to make key-based authentication incredibly convenient and flexible. It offers various security benefits over passphrase-free keys.

#13: Chroot SSHD (Lock Down Users To Their Home Directories)

By default users are allowed to browse the server directories such as /etc/, /bin and so on. You can protect ssh, using os based chroot or use special tools such as rssh. With the release of OpenSSH 4.8p1 or 4.9p1, you no longer have to rely on third-party hacks such as rssh or complicated chroot(1) setups to lock users to their home directories.

#14: Use TCP Wrappers

TCP Wrapper is a host-based Networking ACL system, used to filter network access to Internet. OpenSSH does supports TCP wrappers. Just update your /etc/hosts.allow file as follows to allow SSH only from :

sshd :

#15: Disable Empty Passwords

You need to explicitly disallow remote login from accounts with empty passwords, update sshd_config with the following line:

PermitEmptyPasswords no

#16: Thwart SSH Crackers (Brute Force Attack)

Brute force is a method of defeating a cryptographic scheme by trying a large number of possibilities using a single or distributed computer network. To prevents brute force attacks against SSH, use the following softwares:

* DenyHosts is a Python based security tool for SSH servers. It is intended to prevent brute force attacks on SSH servers by monitoring invalid login attempts in the authentication log and blocking the originating IP addresses.
* Explains how to setup DenyHosts under RHEL / Fedora and CentOS Linux.
* Fail2ban is a similar program that prevents brute force attacks against SSH.
* security/sshguard-pf protect hosts from brute force attacks against ssh and other services using pf.
* security/sshguard-ipfw protect hosts from brute force attacks against ssh and other services using ipfw.
* security/sshguard-ipfilter protect hosts from brute force attacks against ssh and other services using ipfilter.
* security/sshblock block abusive SSH login attempts.
* security/sshit checks for SSH/FTP bruteforce and blocks given IPs.
* BlockHosts Automatic blocking of abusive IP hosts.
* Blacklist Get rid of those bruteforce attempts.
* Brute Force Detection A modular shell script for parsing application logs and checking for authentication failures. It does this using a rules system where application specific options are stored including regular expressions for each unique auth format.
* IPQ BDB filter May be considered as a fail2ban lite.

#17: Rate-limit Incoming Port # 22 Connections

Both netfilter and pf provides rate-limit option to perform simple throttling on incoming connections on port # 22.

Iptables Example

The following example will drop incoming connections which make more than 5 connection attempts upon port 22 within 60 seconds:

$IPT -I INPUT -p tcp --dport ${ssh_port} -i ${inet_if} -m state --state NEW -m recent --set
$IPT -I INPUT -p tcp --dport ${ssh_port} -i ${inet_if} -m state --state NEW -m recent --update --seconds 60 --hitcount 5 -j DROP

Call above script from your iptables scripts. Another config option:

$IPT -A INPUT -i ${inet_if} -p tcp --dport ${ssh_port} -m state --state NEW -m limit --limit 3/min --limit-burst 3 -j ACCEPT
$IPT -A INPUT -i ${inet_if} -p tcp --dport ${ssh_port} -m state --state ESTABLISHED -j ACCEPT
$IPT -A OUTPUT -o ${inet_if} -p tcp --sport ${ssh_port} -m state --state ESTABLISHED -j ACCEPT
# another one line example
# $IPT -A INPUT -i ${inet_if} -m state --state NEW,ESTABLISHED,RELATED -p tcp --dport 22 -m limit --limit 5/minute --limit-burst 5-j ACCEPT

See iptables man page for more details.

*BSD PF Example

The following will limits the maximum number of connections per source to 20 and rate limit the number of connections to 15 in a 5 second span. If anyone breaks our rules add them to our abusive_ips table and block them for making any further connections. Finally, flush keyword kills all states created by the matching rule which originate from the host which exceeds these limits.

table persist
block in quick from
pass in on $ext_if proto tcp to $sshd_server_ip port ssh flags S/SA keep state (max-src-conn 20, max-src-conn-rate 15/5, overload flush)

#18: Use Port Knocking

Port knocking is a method of externally opening ports on a firewall by generating a connection attempt on a set of prespecified closed ports. Once a correct sequence of connection attempts is received, the firewall rules are dynamically modified to allow the host which sent the connection attempts to connect over specific port(s). A sample port Knocking example for ssh using iptables:

$IPT -N stage1
$IPT -A stage1 -m recent --remove --name knock
$IPT -A stage1 -p tcp --dport 3456 -m recent --set --name knock2

$IPT -N stage2
$IPT -A stage2 -m recent --remove --name knock2
$IPT -A stage2 -p tcp --dport 2345 -m recent --set --name heaven

$IPT -N door
$IPT -A door -m recent --rcheck --seconds 5 --name knock2 -j stage2
$IPT -A door -m recent --rcheck --seconds 5 --name knock -j stage1
$IPT -A door -p tcp --dport 1234 -m recent --set --name knock

$IPT -A INPUT -p tcp --dport 22 -m recent --rcheck --seconds 5 --name heaven -j ACCEPT
$IPT -A INPUT -p tcp --syn -j doo

* fwknop is an implementation that combines port knocking and passive OS fingerprinting.
* Multiple-port knocking Netfilter/IPtables only implementation.

#19: Use Log Analyzer

These tools make your log reading life easier. It will go through your logs for a given period of time and make a report in the areas that you wish with the detail that you wish. Make sure LogLevel is set to INFO or DEBUG in sshd_config:

LogLevel INFO

#20: Patch OpenSSH and Operating Systems

It is recommended that you use tools such as yum, apt-get, freebsd-update and others to keep systems up to date with the latest security patches.

Other Options

To hide openssh version, you need to update source code and compile openssh again. Make sure following options are enabled in sshd_config:

# Turn on privilege separation
UsePrivilegeSeparation yes

# Prevent the use of insecure home directory and key file permissions
StrictModes yes

# Turn on reverse name checking
VerifyReverseMapping yes

# Do you need port forwarding?
AllowTcpForwarding no
X11Forwarding no

# Specifies whether password authentication is allowed. The default is yes.
PasswordAuthentication no

Verify your sshd_config file before restarting / reloading changes:
# /usr/sbin/sshd -t

Tighter SSH security with two-factor or three-factor (or more) authentication.

Read the rest of this entry...

Bookmark and Share My Zimbio

Thursday, August 6, 2009

How to kill defunct processes

If you have been an system administrator for Solaris for sometime you should be familiar with such processes. Basically a defunct or much more known as a zombie process is a process that has completed execution but still has an entry in the process table, this entry being still needed to allow the process that started the zombie process to read its exit status. The term zombie process derives from the common definition of zombie—an undead person. In the term's colorful metaphor, the child process has died but has not yet been reaped.

Whenever I encounter this zombies I normally would kill the parent process that spawned it and much worst if this doesn't work I might have to restart the whole server. Well I've found another option ( befroe you start rebooting the machine).

There's a little less known command to try and kill these defunct/zombie process and that is the command preap.

From the man pages:

preap - force a defunct process to be reaped by its parent

preap [-F] pid...

A defunct (or zombie) process is one whose exit status has
yet to be reaped by its parent. The exit status is reaped
via the wait(3C), waitid(2), or waitpid(3C) system call. In
the normal course of system operation, zombies may occur,
but are typically short-lived. This may happen if a parent
exits without having reaped the exit status of some or all
of its children. In that case, those children are reparented
to PID 1. See init(1M), which periodically reaps such

An irresponsible parent process may not exit for a very long
time and thus leave zombies on the system. Since the operat-
ing system destroys nearly all components of a process
before it becomes defunct, such defunct processes do not
normally impact system operation. However, they do consume a
small amount of system memory.

preap forces the parent of the process specified by pid to
waitid(3C) for pid, if pid represents a defunct process.

preap will attempt to prevent the administrator from
unwisely reaping a child process which might soon be reaped
by the parent, if:

o The process is a child of init(1M).

o The parent process is stopped and might wait on the
child when it is again allowed to run.

o The process has been defunct for less than one minute.

So to kill a defunct process you can try:

server# ps -ef| grep -i defunct
oracle 23650 22802 0 - ? 0:01
oracle 23657 22802 0 - ? 0:01
oracle 23580 22802 0 - ? 0:00
oracle 23924 16560 0 - ? 0:00
oracle 23750 22802 0 - ? 0:01
oracle 23928 16363 0 - ? 0:00
oracle 23915 17114 0 - ? 0:00
oracle 23940 20910 0 - ? 0:00
oracle 23863 21896 0 - ? 0:00

server# ps -ef| grep -i defunct |awk {'print $2'}|xargs preap
23650: killed by signal KILL
23657: killed by signal KILL
23580: killed by signal KILL
23924: exited with status 141
23750: killed by signal KILL
23928: exited with status 141
23915: exited with status 141
23940: killed by signal KILL
23863: killed by signal KILL
23921: exited with status 141
23912: exited with status 141
23652: killed by signal KILL
23889: exited with status 141
23752: killed by signal KILL
23931: exited with status 141
23925: exited with status 141
23936: exited with status 141
23916: exited with status 141
23784: killed by signal KILL
23744: killed by signal KILL
23656: killed by signal KILL
preap: cannot examine 6343: no such process
23926: exited with status 141
23651: killed by signal KILL
23631: killed by signal KILL
23922: exited with status 141
23654: killed by signal KILL
23781: killed by signal KILL
23933: exited with status 141
23923: exited with status 141
23790: killed by signal KILL
24011: exited with status 141
23938: exited with status 141
23634: killed by signal KILL
23907: exited with status 141
23864: exited with status 141
23908: exited with status 141
23883: exited with status 141
23812: killed by signal KILL
23765: killed by signal KILL
23906: exited with status 141
23910: exited with status 141
23871: exited with status 141
23653: killed by signal KILL
23902: killed by signal KILL
23782: killed by signal KILL
23743: killed by signal KILL
server# ps -ef| grep -i defunct

If this still fails to kill them then you have to restart the mother process or the server itself.

Read the rest of this entry...

Bookmark and Share My Zimbio

Saturday, August 1, 2009

ZFS Tip: Comparison of SVM mirroring and ZFS mirroring

Most of us are familiar with SVM (a.k.a. Solaris Volume Manager, Disksuite, Onlne Disksuite, ODS, Solstice Disksuite, SDS, etc)

Under SVM we build "metadevices" from one or more disks. We can then mirror equal sized metadevices to create a "metamirror".

In this example we have two 30GB metadevices, one happens to be a concatenation of dissimilar disks, the other is a striped over equal sized disks.

The two metadevices are mirrored together to create a 30GB metamirror.
If any disk fails, the entire submirror will go offline but the metamirror will remain online.

|30GB metamirror |
| ----------------------------------------------------------- |
| |30GB submirror built from concatenation of two disks | |
| | ------------- ------------ | |
| | | 10G | | 20G | | |
| | | disk | + | disk | | |
| | ------------- ------------ | |
| ----------------------------------------------------------- |
| |
| ----------------------------------------------------------- |
| |30G submirror striped over three 10GB disks | |
| | ------------- ----------- ------------ | |
| | | 10G | | 10 | | 10G | | |
| | | disk | : | disk | : | disk | | |
| | ------------- ------------ ------------ | |
| ----------------------------------------------------------- |
| |

In a ZFS pool we mirror inside the vdevs.

In the example below, we have a 45GB ZFS pool. Data in the pool is spread dynamically over three independent vdevs.

Two of the vdevs are mirrored, the third is unmirrored.

There is no requirement for the vdevs to be the same size.

The disks (or partitions) within a single vdev should be the same size or some disk space will go to waste.

If a disk in one of the mirrored vdevs fails, the vdev will enter an unmirrored state but there will be no loss of data.

If the disk in the unmirrored vdev fails, the entire vdev will fail and this will cause the entire pool to fail.
|45GB ZFS pool |
| ---------------------- ---------------------- ----------------------- |
| | 10G mirrored vdev | | 20G mirrored vdev | | 15G unmirrored vdev | |
| | ------------- | | ------------- | | | |
| | | 10G | | | | 20G | | | ------------ | |
| | | disk | | | | disk | | | | 15G | | |
| | ------------- | : | ------------- | : | | disk | | |
| | ------------- | | ------------- | | ------------ | |
| | | 10G | | | | 20G | | | | |
| | | disk | | | | disk | | | | |
| | ------------- | | ------------- | | | |
| ---------------------- ---------------------- ----------------------- |
| |

In the real world we would probably not create a pool where some vdevs are mirrored and some are not.

If we have more than three disks in a vdev we have a choice of a "three-way mirror" or we could use RAIDZ. RAIDZ is an improvement over RAID-5.

If we are using LUNs from the SAN which are already built in a RAID array, there may be little value in using mirroring or RAIDZ within a vdev.

Read the rest of this entry...

Bookmark and Share My Zimbio