LoReM iPsUM awk

Did you ever get a list of stuff, like email addresses or domain names or something, and the person who gave it to you did crazy capitalization?  Some are all lowercase, some are uppercase, but the only thing that's consistent is there's no commonality.

awk to the rescue again.

Let's assume you have a list of stuff you've copied and pasted into a text file called delete.txt.  This one-liner will amend your ailment:

 cat delete.txt |awk '{print tolower ($0)}'

Voila.

That's what she sed.

More sed magick today in the salt mines.

Here was my issue.  I was moving dozens of domains from their position on a cluster server, and moving over to another machine that was to take over as the cluster 'master'.  Since the server they were on was a live server, I couldn't just move the IP addresses without blowing up all of those sites, and DNS would tell me that the old server was working fine; how am I going to test the sites on the new server before I flip the switch?

Well, edit the hosts file on my local machine of course.  Hrm.  There might be 100 sites that I'd need to change the IP for in my hosts file; that's going to suck.

Not really, because my friends sed and awk can make quick work of this.

Assuming:

A) I have 2 servers that have multiple vhost entries in a file called client-vhost.conf
B) the server IP addresses bound to apache are x.x.x.x (server1) and y.y.y.y (server2)
C) the public document root for the websites have subfolders that represent the domain name:

(ie. drwxrwxr-x 7 www www 4096 Dec 1 2011 example.com)

D) we've already synced the data and config files so they are identical on both machines.
E) 5.X CentOS install with configs in /etc/httpd/conf/ and data folder of /var/www/

Since server1 has been up and running serving web sites for a period of time, all of our work is going to be on server2.  Edit the client-vhost.conf file, you're going to need to do the first part by hand.  Change the IP apache is listening on.  We don't want to pull out the old entries, because server2 is going to take the IPs from server1. Comment out the x.x.x.x lines and insert the y.y.y.y lines:

#Listen x.x.x.x:80
#NameVirtualHost x.x.x.x
Listen y.y.y.y:80
NameVirtualHost y.y.y.y

Then save.

Now we're going to edit all of the vhost entries, there's a lot- so sed will do all the lifting for us.  From the command-line type:

sed -i 's/x.x.x.x/x.x.x.x\ y.y.y.y/g' /etc/httpd/conf/client-vhost.conf

That will replace the search string 's/----- with the global replace string /--------/g'.  We put a space in between the server1 IP address and the server2 IP address, so we need to escape the space with a '\'.

Now we need to get the new IP into my hosts file with all the domain names.  How can I get the columns displayed easily with the new IP and a space followed by the domain name?  Like this:

ll /var/www/ | grep com | awk '{print "y.y.y.y " $9}'

What I did there- 'll' is kinda like 'ls -la' and displays 9 columns; I'm greping 'com' for the *.com reference because there are potentially folders in there that aren't published domains, and I don't care about those; awk takes the piped information and prints "y.y.y.y " (<- with a space before the last ' " ') then $9 or the ninth column, which is the directory name.  This gave me a nice two-column list I can copy/paste into my c:\windows\system32\drivers\etc\hosts file (right-click notepad and Open as Administrator if you're using Windows post-XP) that looks like this:

y.y.y.y example1.com
y.y.y.y example2.com
y.y.y.y example3.com

Now I can just paste the domains into my browser and make sure they work on the new box before making any potentially destructive changes.  Which was a good call, because the real-life example this post was based off of had a few things missing in the server configuration.

iptables, oh no you didn't..

Being a systems administrator for a large number of linux systems that are primarily web and email servers, I get to see a lot of attacks on different services.

One of our boxes (A) was brought down from another box for the same client (B).  A. was running httpd and B. was running sendmail.  After examining the logs and determining that B. was the culprit, an examination of the logs on B. showed a particular IP with a large number of simultaneous connections.  His vector was to do dictionary/name email blast to the domain, then when he got a valid user, he would try to authenticate via another dictionary attack.  This was spawning a large number of sendmail processes that ended up slowing down the box considerably.

Performing a whois on the IP address showed us that the hacker was operating from another country that most likely was not a customer of our client.  

whois X.X.X.X

In order to keep the box and network alive, we decided to block the IP address on the machine's firewall.  We didn't want to add the IP to be there on reboot, in case it was a DHCP address that might be legitimate in the future, but we can insert the IP address inline, which will block it until the iptables is restarted.  If this IP comes through again in the future, we would add it to the /etc/sysconfig/iptables and reload the configuration.

This one-liner will keep any traffic from the 'bad' address from doing anything further on this box:

iptables -I INPUT -s X.X.X.X -j DROP

As soon as you hit enter, that attack is over.  If you realize you typed in the wrong IP address, and you want to remove the inline rule you just created, change the -I to a -D like this:

iptables -D INPUT -s 208.122.195.120 -j DROP

If you wanted to block this IP permanently, simply add this line to the /etc/sysconfig/iptables:

-A INPUT -s X.X.X.X -j DROP

Then reload your iptables config.

I BASHed the terminal and sed 's/ex...'

Like MySQL, BASH scripting is something I've always done frequently, but very light as far as complexity and depth.

Batching commands is easy, but now I'm coming into a new phase of my career where scripting is crucial to the day-to-day operations.  I'm finding help in the form of co-workers and tldp.org.  The following is a sanitized version of the check and insert I was asked to create that will turn on fasttcp on CentOS/RHEL, but not if the kernel is "2.6.18-238.9.1.el5.fsl_2.0.2.1"

So we need to:
a) Check kernel to exclude a certain build.
      i. (if yes) Insert 'kernel*' to the exclude on yum.conf
     ii. Report if the yum.conf was modified or not.
b) download and install
c) chkconfig on resources

I'm a total noob at scripting, so I made sure I didn't break anything on the system; I created the script as a non-root user and copied the yum.conf to that user's home folder with a unique name.

cp /etc/yum.conf /home/dave/dave-practice-yum.conf

First, the beginning of any (good) shell script calls the shell you're working in.  I'm from BSD most-recently, so !/bin/sh is what I would generally use, but now that I'm a CentOS guy at my new job, !/bin/bash is the shell for me.

#!/bin/bash

Defining variables: I need to find out if the kernel is the mystical kernel that isn't supposed to get updated, so I'll establish the KRNL variable.  I also am going to insert 'kernel*' into the 'exclude=' line of the yum.conf if it is a match, so I will also define the EXCL variable to make sure I'm not over-writing an already-established 'exclude='.

KRNL=`uname -r`
EXCL=`grep "exclude=" dave-practice-yum.conf|cut -d= -f2`

Now we write in the actual check with an IF statement:

if [ $KRNL = 2.6.18-238.9.1.el5.fsl_2.0.2.1 ]
then
        if [ -z $EXCL ]
        then
                sed -i 's/exclude=/exclude=kernel*/' dave-practice-yum.conf
                echo "yum.conf modified."
        else
                sed -i '/exclude=/s/$/,kernel*/g' dave-practice-yum.conf
        fi
else
        echo "No modification to yum.conf"
fi

The else in the nested condition is pretty ugly.  I didn't want to over-write if there were already any excludes, so I appended the 'kernel*' to the 'exclude=' if nothing were present; if there is something present on the exclude line when the script runs, it will simply put a ',kernel*'.  This works, but if 'kernel*' is there, it will be repeated.

Then we download the source package from the internal repository, and chkconfig the services on (in this case FASTTCP).

/usr/sbin/wget -qO - http://server/files/fasttcp/ftcp_rc.tar|tar xf - -C /
/sbin/chkconfig --add fasttcp
/sbin/chkconfig --levels 2345 fasttcp on
/etc/init.d/fasttcp start

Remarkably, this script works; thanks in large part to my co-workers, and James in particular.  Below, find the script in its entirety (without blogspot formatting).

#!/bin/bash

KRNL=`uname -r`

EXCL=`grep "exclude=" dave-practice-yum.conf|cut -d= -f2`

if [ $KRNL = 2.6.18-238.9.1.el5.fsl_2.0.2.1 ]

then

        if [ -z $EXCL ]

        then

                sed -i 's/exclude=/exclude=kernel*/' dave-practice-yum.conf

                echo "yum.conf modified."

        else

                sed -i '/exclude=/s/$/,kernel*/g' dave-practice-yum.conf

        fi

else echo "No modification to yum.conf"

fi

/usr/sbin/wget -qO - http://server/files/fasttcp/ftcp_rc.tar|tar xf - -C /

/sbin/chkconfig --add fasttcp

/sbin/chkconfig --levels 2345 fasttcp on

/etc/init.d/fasttcp start

A funny thing happened on the way to the Internet...

Has anybody else noticed the kooky stuff that's been going on pretty much globally?  We have experienced crazy DNS disappearances, terrible latency, dropped VPN connections, web servers unresponsive, and that's just the beginning.  I'm not just talking about the film studio, or my home-based business; I'm also talking about things like GoDaddy and Network Solutions (ok, I know, Network Solutions was hardly ever 'fast' to begin with).  Something Big and most certainly Bad is going on.

Terrible things are going on and I'm concerned that we are about to find out what.  Is it a DDoS at our backbones?  Are they poisoning DNS?  Who are they?  Will someone make me a sandwich?

Format a USB disk with ext3 in Red Hat style systems.

We have some Red Hat boxes that do environmental effects on some of the motion pictures we do.  These guys need to archive some of their stuff onto USB disks so they can transport them or give them to clients, etc.  The problem is that they usually come formatted with FAT32 or something, and the files can be greatly in excess of 2.0GB.  So we put ext3 on.

First, see if the USB disk mounted itself.  Sometimes Red Hat does this.  Use df or mount to see if it's there.
[root@discreet ]# df

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda3            151668928  43941540 100023016  31% /
/dev/sda1               101086     13285     82582  14% /boot
none                   3928520        16   3928504   1% /dev/shm
/dev/sdb1            721075720     70584 684376536   1% /mnt/usbdisk1

Unmount /mnt/usbdisk1.
[root@discreet ]# umount /mnt/usbdisk1
then fdisk it (must be root!)
 [root@discreet ]# /sbin/fdisk /dev/sdb

The number of cylinders for this disk is set to 91201.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Input a p to list current partitions.
Command (m for help): p

Disk /dev/sdb: 750.1 GB, 750156374016 bytes
255 heads, 63 sectors/track, 91201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1       91201   732572001    c  W95 FAT32 (LBA)  ### FAT DOES NOT ALLOW >2GB FILES

Delete the FAT32 partition with d.
Command (m for help): d
Selected partition 1


Do another p to display the partition list after you have deleted partition 1.
Command (m for help): p       

Disk /dev/sdb: 750.1 GB, 750156374016 bytes
255 heads, 63 sectors/track, 91201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System

We're going to hit n for new partition.
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p           ###### P FOR PRIMARY PARTITION


Partition number (1-4): 1
First cylinder (1-91201, default 1):
Using default value 1       ##### PRESS ENTER FOR CYLINDER 1 TO BE START OF PARTITION
Last cylinder or +size or +sizeM or +sizeK (1-91201, default 91201):   #### JUST PRESS ENTER FOR DEFAULT PARTITION TO TAKE UP WHOLE DRIVE
Using default value 91201

Print the partition table one more time to make sure it's ext3
Command (m for help): p      

Disk /dev/sdb: 750.1 GB, 750156374016 bytes
255 heads, 63 sectors/track, 91201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1       91201   732572001   83  Linux  

Write and quit with wq.
Command (m for help): wq               ####### W FOR WRITE, Q FOR QUIT
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Make the new file system.

[root@discreet ]# /sbin/mkfs /dev/sdb1

mke2fs 1.35 (28-Feb-2004)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
91586560 inodes, 183143000 blocks
9157150 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
5590 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000
Writing inode tables: done
Writing superblocks and filesystem accounting information: done

Mount your new partition where you like and copy large files to it.

Captain, I'm giving her all she's got.

I need to use Linux specifically for one program my friend some asshole wrote.  The machine it's on has only 512 MB of RAM and just running the GUI on Ubuntu 10.4 LTS puts it 20%-30% into the swap file.

The quickest and surest way to free up the much needed RAM is to disable the gdm.  You can do this by renaming/breaking the script that init loads at boot time.

$ sudo mv /etc/init/gdm.conf /etc/init/gdm.broke
and reboot.

You may send me a check for the RAM you save.  Contact me for details.

A trail of breadcrumbs.

This is an accounting of 'first steps' when building a virtual server with FreeBSD.  Documentation on these steps will be sparse.

1. Install host system
2. Configure system files

Configure: /etc/rc.conf

hostname="HOSTNAME"

## this section is hardware/system
moused_enable="YES"
sshd_enable="YES"
zfs_enable="YES"

## this section is the aggregated network interfaces
ifconfig_em0="up"
ifconfig_em1="up"
ifconfig_em2="up"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport em0 laggport em1 laggport em2"
ipv4_addrs_lagg0="10.0.10.149/24"
defaultrouter="10.0.10.1"

## this section is NFS and file sharing
nfs_client_enable=YES
syslogd_flags="-ss"
rpcbind_enable="YES"
nfs_server_enable="YES"
mountd_flags="-r"

## configure jails here
ezjail_enable="YES"
## first jail IP
ifconfig_lagg0_alias0="inet 10.0.10.148/32"

Configure DNS: /etc/resolv.conf
nameserver 8.8.8.8  ## 8.8.8.8 is google DNS, put yours or something that works here

3. Create ZFS pool and move selected folders from boot drive to ZFS array
# zpool create tank mirror ad8 ad12
# zfs create tank/src
# zfs create tank/obj
# zfs create tank/jails

# zfs create tank/ports

# zfs set mountpoint=/usr/src tank/src

# zfs set mountpoint=/usr/obj tank/obj

# zfs set mountpoint=/usr/jails tank/jails

# zfs set mountpoint=/usr/ports tank/ports

4. download source code and ports tree

# pkg_add -r cvsup-without-gui

# cp /usr/src/share/examples/cvsup/standard-supfile /root

# rehash

# vi /root/standard-supfile

############ change the CHANGETHIS line to something like:

*default host=cvsup3.us.FreeBSD.org

# cvsup /root/standard-supfile

# portsnap fetch extract

5. Update your system

6. Make your server do stuff.

ZFS is the best file system ever.

NOTE: I will be using the term 'RAID' very loosely later in this post.  When I describe later RAID 1 and RAID 10 style arrays I am using these terms to illustrate how ZFS works in a manner most people are familiar with.

ZFS takes everything I used to know about partitions, volumes and disk space and spins it up in a blender.  Unlike the traditional file system where the size of your disk or array must be specified at the beginning, ZFS allows you to simply pick your parity and add devices as your needs for speed or space increase.  ZFS is similar to RAID but has many advantages over traditional RAID.  I'm not going to go into detail as there are many Sun/Oracle Solaris developer articles on the subject.  You can start reading about it here.

Assuming you have FreeBSD installed and two identical disks that have not been partitioned or sliced in any way named ad8 and ad10 we can get our first disk pool going with zpool.  I'm going to call my disk pool tank because most people do.

# zpool create tank mirror ad8 ad10

That created a RAID 1 style vdev, or virtual device mounted at location /tank.  Note there are other types of vdevs; I will not go into detail on them, but other types include raidz, raidz2 and no parity.   We will be able to use the amount of space equal to one of the disks, and we will have excellent read I/O and acceptable write I/O.  We will be able to continue using the pool in the event of one of the disks failing.

To ensure the pool is mounted at boot time, echo this statement to /etc/rc.conf:

# echo 'zfs_enable=YES'>>/etc/rc.conf

We can see the health of the pool by typing:

# zpool status -v

Let's assume now that we have filled the tank up and we need to add more space.  The best way to do this is to purchase two more disks of the same capacity we had on ad8 and ad10, and add them to the the pool as another mirrored vdev.  Assuming these new disks are identified as ad12 and ad14, type:

# zpool add tank mirror ad12 ad14

Now when you do a status -v on your zpool you will see two mirrors listed.

So wait, that's it?  We just doubled the capacity of /tank?  Yes.  Not only did we double the capacity, but we also doubled the speed.  We effectively converted our "RAID 1" into a "RAID 10".  This will give us excellent read and write I/O, and the ability to continue working if any one disk from any one vdev fails.  If you want it bigger or faster, keep adding vdevs.

We will periodically want to scrub the pool to make sure there aren't any errors.  To do this type:

# zpool scrub tank

This will queue up a disk scrub which we can monitor with the zpool status -v.  You might want to add this to a weekly cron job.  If any errors are discovered, they can be easily cured via the metadata ZFS keeps on the parity disks with the clear switch.

# zpool clear tank

ZFS keeps a lot more metadata than traditional hardware RAID systems, making it much safer for data archiving.  It also allows you to take a snapshot of the data.  This will allow you to retrieve deleted files or rollback to a specific period of time.  Snapshots are stored in the root of the filesystem in a .zfs folder.  For example, /tank/.zfs/snapshot.  To take a snapshot, type:

# zfs snapshot tank@19may2011

Notice there is no leading '/'; you will now have a folder named /tank/.zfs/snapshot/19may2011/ with what looks like everything you had in there before.  In the event that you need to restore something from a snapshot folder, simply cp it to where it was.  For example:

# cp /tank/.zfs/snapshot/19may2011/my_song.mp3 /tank/

Your missing/overwritten file will be restored to the state it was at that snapshot.

Move data up to 6X faster than you used to with LACP 802.3ad.

If you have a Cisco or other programmable switch capable of LACP 802.3ad then you're most of the way to having faster data transfer than you may have thought possible.  The simplicity of this process still throws me for how powerful it is.  LACP is simply bonding two or more network interfaces at the switch and at the computer so that they act as a single virtual network interface.  This increases the speed and availability by two, three, four or up to six times.  It is not recommended to use more than six physical interfaces in one Link Aggregate.

FreeBSD refers to the link aggregate as laggN where N is the ID of the virtual interface.  For example your first lagg would be lagg0 followed by lagg1 and so on.  The server I am working on is a fresh install of FreeBSD 8.2 amd64 with four physical network interfaces em0 em1 em2 and em3.  It is going to be the permanent replacement of a virtual server that currently has three aggregated network interfaces.  I am using a Cisco 3750 in this specific case, but I have used this same process on a variety of other Cisco switches.  We're going to do the switch first, then we'll set up the server.  This example assumes you are using Gigabit ports 0/10 through 0/13.

After you get into enable mode on the switch type:

# config t
(config)# int range gi0/10-13
(config-if-range)# channel-group 1 mode active
(config-if-range)# channel-protocol lacp
(config-if-range)# exit
(config)# exit

Then type show run to make sure your new interfaces have the new channel- switches in them.

interface GigabitEthernet0/10
 description admin
 switchport access vlan 20
 switchport mode access
 channel-protocol lacp
 channel-group 1 mode active
 spanning-tree portfast

The group number is pretty arbitrary; you can make the number whatever you want for documentation.  Just make sure it's not already used somewhere else on that switch.  Sometimes I have issues getting the channel-protocol line to stick when I choose an int range instead of calling the interface separately.  If this happens to you, simply go back into config t and call each interface separately, give it the channel-protocol line and exit back to config t for the next one, like this:

# config t
(config)# int gi0/10
(config-if)#channel-protocol lacp
(config-if)#exit
(config)#int gi0/11
(config-if)#channel-protocol lacp
(config-if)#exit
(config)#int gi0/12

..And so on.

The switch is done.  We're going to add a few lines to our /etc/rc.conf file and that'll do it for the server.  Pull up /etc/rc.conf with an editor and add:

ifconfig_em0="up"
ifconfig_em1="up"
ifconfig_em2="up"
ifconfig_em3="up"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport em0 laggport em1 laggport em2 laggport em3"
ipv4_addrs_lagg0="10.0.10.149/24"
defaultrouter="10.0.10.1"

The ifconfig_lagg0="laggproto lacp laggport em0 laggport em1 laggport em2 laggport em3" entry is all one line.

When you reboot or activate your new network stack your server will now listen on all four network interfaces making your data transfers four times faster than they used to be.  When you're satisfied that everything is good, go back to your Cisco and copy run start in enable mode to ensure you keep your link aggregation after a switch reboot.


Make sure your disk I/O is fast enough to handle the new speeds by enabling ZFS on your server.  This example makes the network so fast that there is no way you could possibly feed information to it from disk fast enough to use it without some major arrays.  

Using FreeBSD ports effectively.

When I first started with FreeBSD I wasted a lot of time.  I knew where I wanted to go, but not exactly how I wanted to get there.  One thing I did know was using the FreeBSD Ports Collection was definitely the way to go to install most software as it handled dependencies and lots of other issues automatically.

Occasionally, I would find myself installing programs with a huge list of dependencies; if any of these programs required configuration, the whole process would stop and ask you to configure it.  If you have a long list of dependencies, it might not be viable to sit and watch the console during the compilation of all of the ports as this could take hours depending on how much software is to be installed.  A switch for 'make' takes this problem into consideration and pre-configures nearly all dependencies before starting anything.

For example, if you were to install the 'www/apache22' web server and wanted to increase the possibility of an automated install, use the 'config-recursive' switch, like this:

# whereis apache22

apache22: /usr/ports/www/apache22

# cd /usr/ports/www/apache22

# make config-recursive install clean

This will go through the list of dependencies and bring up the 'make config' page for any that require tuning.  This can save a lot of time.

After you install anything via ports, run 'rehash' to make FreeBSD re-read the the files in the path so it knows that your new program is available.

A similar switch for the 'portupgrade' command is '-c'.  If you're updating the whole system and you want to pre-configure the updates try this:

# portupgrade -c -a

If you find that you no longer need a program installed and you want to ensure that no extraneous ports are left over in the system, remove it with 'pkg_deinstall' using the '-R' switch, like this:

# pkg_deinstall -R audio/amarok

It will uninstall 'amarok' and every dependent port for it that is not being used by another port.  If another program requires the same dependency 'pkg_deinstall' will not remove that dependency.

If you have a port that is affected by a security problem but you absolutely must install the port anyway, you can force it with the 'DISABLE_VULNERABILITIES=true' switch, like this:

# make DISABLE_VULNERABILITIES=true config-recursive install clean

Now the port affected by the security problem will not be blocked from install.

The Playstation Network is (back) online.

I rented TRON for the PS3 last night because Final Fantasy XIII was checked out.  I played the single-player campaign mode last night and didn't get out of Chapter 1 even with my saved progress from the last time I rented it.  This morning I fired it up to get some Light Cycle action in the scene after the ZUSE scene when I noticed the system update from this morning.

I updated the PS3 software with the 3.61 which went pretty quickly; I had to change my PSN password- good thinking Sony, and I logged on to a Quick Match.

I was on the Blue Team and Orange had already gotten the tank.  I captured the tank flag from the Orange Team and sat down for some serious damage.  The arena is a Light Cycle track with 5 'flags' to capture.  The team that holds more flags for the longest wins.  Flag #3 is right in the center of the map.  I shambled my way to Flag #3 as my tank piloting skills apparently aren't that great.  I did manage to blast a few Orange fellows off their Light Cycles and drive over a few pedestrian-based Orange Team.

I spent the next 8 consecutive minutes at Flag #3 which after I captured it was blue for the remainder of the game.  I had good Line of Sight on most of the map so I was able to keep the orbiting enemies at bay with suppressive fire.  I got some trophies and top score first match back for the PSN!

Yay.

Viva la PlaystationNetwork.

A daemon for your every command: cron

Cron is an excellent way to do regular tasks.  It uses 'vi' as its editor, and the only real requirement is that you use full path names like:

/usr/sbin/portsnap

Instead of:

portsnap

From the root user type:

# crontab -e

This will bring up the schedule to be edited.  The schedule follows the columns of minutes, hours, days, months and day of the week and is very customizable.  FreeBSD allows the switches '@daily', '@weekly', etc, not all systems will allow those.  All of the following examples are single lines.

For example, to schedule the download and update of the new ports tree on a FreeBSD system at 4:41am every day, add:

41 4 * * * /usr/sbin/portsnap -I cron &&/usr/sbin/portsnap update

If you wanted to update the ports you set up with ezjail at 5:05am, add:

5 5 * * * /usr/local/bin/ezjail-admin update -P

To zip and archive your system config files at midnight add this on a single line:

@daily /usr/bin/tar zcvf /tank/backup/etc.tar.gz /etc&&/usr/bin/tar zcvf /tank/backup/usrlocaletc.tar.gz /usr/local/etc

Back up your mysql database every night at 3:00am, add:

0 3 * * * /usr/local/bin/mysqldump -hlocalhost -uintranetguy -pd0ntTe3ll intranet > /var/sqlbackup/intranet.sql

And don't forget your weekly scrub for your ZFS array:

@weekly /sbin/zpool scrub tank

Just like 'vi' you can Esc to command mode and exit with a ':' followed by 'wq' and Enter.

ezjail is easy, so virtualize.

FreeBSD is an efficient operating system.  It offers great scalability as well in the form of virtual environments referred to as jails.  Jails offer protected areas for services to run that would not normally be able to exist on the same machine.  They offer separate virtual machines, however they do not need to start their own kernel, they simply run processes off the host system's kernel making each virtual machine little more overhead than the services you're starting inside them.

Installing from ports:

# whereis ezjail

ezjail: /usr/ports/sysutils/ezjail

# cd /usr/ports/sysutils/ezjail

# make install clean

This is a fast process.  ezjail is very light. You can rehash when you're done so you can run it immediately.  The following commands will create and modify the config files under /usr/local/etc/ezjail.

If this is a fresh install and you've not done a buildworld yet, you can install jail binaries.  If you have your /usr/src straightened out all up to date with patches and everything use the second method.
Fresh install method 1:

# ezjail-admin install

If you're all set on the /usr/src and you've done makeworld already method 2:

# ezjail-admin update -p -i

The '-p' will give your jails ports and the '-i' skips the build/make for the world.

There are a few details we need to establish before we make the first jail.  You will need a separate IP address for each jail you create.  We're going to assume you know how to identify your network interface; contact me if you need to know how to do this as it would make an excellent post.  We will need to alias the new IP to the network interface.  To make it stick on reboot, we add the alias line to /etc/rc.conf.  The overall process of making a jail is quite easy.  Simply modify a few files and start creating jails.  First turn syslog and sshd to listen only on the host system's IP address.

Assuming you have an IP of 192.168.1.50 on the host system network interface em0 and the IP block of 192.168.1.51-55 is free (at least 51 for this example).

Echo the following statement to /etc/rc.conf for syslog.

# echo 'syslogd_flags="-ss"'>>/etc/rc.conf

Edit /etc/ssh/sshd_config and add the following line near the top:

ListenAddress 192.168.1.50

If you have other services running on the host system, then you'll have to do something similar to their files.

We're going to commit the IP address to the interface now so we can create the jail, then fix it to boot with it in after that.  Like this:

# ifconfig em0 192.168.1.51/32 alias

Note the subnet is CIDR /32 or 255.255.255.255 as opposed to 255.255.255.0 as you would expect with a vanilla class C network.  The subnet on the jails must be different than that of the host system.

We need the IPs established and for the jails to start at boot.  Edit the file /etc/rc.conf and add:

ezjail_enable=YES

ifconfig_em0_alias0="inet 192.168.1.51/32"

Now our jails will start, and our first jail IP address is specified.  This jail is going to be an FTP server called dudesftp and will live under /usr/jails/dudesftp.

# ezjail-admin create dudesftp 192.168.1.51

A bunch of files will go by starting with /usr/jails.  The jail is now created.  Console in and start setting up your new virtual server.  You will want to set up files like /etc/hosts, /etc/resolv.conf and /etc/rc.conf.  You might also want to turn on sshd and create users for inside the new jail.

# ezjail-admin console dudesftp

That will log you on as root after starting the jail for the first time.  Other useful ezjail-admin commands:

ezjail-admin stop JAILNAME      ## stops the jail JAILNAME
ezjail-admin delete (-w) JAILNAME  ## deletes the jail JAILNAME -w deletes the files that make up the jail as well
ezjail-admin reboot JAILNAME  ## reboots the jail JAILNAME

Mysql your way out of a wet paper bag.

For someone who doesn't have to deal with the subject in depth, I have found myself dealing with mysql very often.  Getting servers migrated and creating services for users requires me to have at least a very basic tool kit for dealing with mysql.

This could be a road map for starting administrators that haven't dealt with mysql.  I never have to deal with tables, but I do need to create databases, make users that have permissions on those databases (and not the whole sql server), back up databases, and restore databases; the following represents my expertise on this subject.

This example assumes a unix-based system with a current (5.X) mysql installation and system root access.  If you want to know how to actually set the server up send me a message; sooner or later I'll post something like that.

1. Creating databases.

Before you can create a database you have to log in to the database server as the root mysql user.  This is not the same as the root user for the host system.  If this is a fresh install, the root account for mysql has no password, and you can omit the '-p' switches.  Mysql has to be running before you can log in.  Each system has different scripts for starting the mysql service, but if you run a 'top' or 'ps aux| grep mysqld' you should be able to see if the 'mysqld' process is running.  To log in type:

mysql -u root -p

The system will then prompt you for the root password.  If you have never set a root password for mysql, leave out the -p.  You'll be greeted with the 'mysql>' prompt instead of your regular system prompt.  You can see what's there by typing the following:

mysql> show databases;

It will most likely show you a list like this:

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| intranet           |
| mysql              |
| performance_schema |
| test               |
+--------------------+
5 rows in set (0.00 sec)

You might not have one that says 'intranet' if you do, then we have something in common.  If you didn't and really wanted one, type:

mysql> create database intranet;

Notice on both commands there is a semi-colon ';' at the end of each command; those are important.

2. Creating users.

In order for the database to be of any use to anyone except the root mysql user, we need to add a mysql user with privileges to that database.  If you're bringing the database over from another machine (we'll get to the restore in a moment) you'll need to make sure that the mysql user name and passwords match exactly.  How we do this is with the following statement:

mysql> grant all privileges on databasename.* to 'mysqluser'@'localhost' identified by 'mysqlpassword';

For our specific example it would look like:

mysql> grant all privileges on intranet.* to 'intranetguy'@'localhost' identified by 'd0ntTe3ll';

That will give the user 'intranetguy' access to write to the database 'intranet' and all of its tables as long as the user logs in from the machine 'localhost' (which usually means it's a process running already on the local box, like a web-server environment would be) with a password of 'd0ntTe3ll'.

It is a good idea to flush the privileges after creating users or databases; then exit the mysql shell.

mysql > flush privileges;
mysql > exit

3. Backing up databases.

I'm a passionate guy, and that passion extends to backing up data.  Back up your databases.  I use 'mysqldump' to make an easy to move sql file that we can use as a backup, or a package to move the database to a new server.  The 'mysqldump' command is run on the host system shell, NOT the mysql> shell.  The statement would look something like this:

# mysqldump -hlocalhost -uintranetguy -pd0ntTe3ll intranet > /path/to/sqlbackupfolder/intranet.sql

Without spaces, the switches are '-h' host - almost always localhost -, '-u' the user you specified earlier, and '-p' the password.  I know what you're thinking: 'I have to type that all the time?' No, we're going to add it to the system cron.  As the system (not mysql) root user, type 'crontab -e' to edit the cron daemon's task list.  This will bring up the cron file in the editor 'vi'; if you don't know how to use 'vi' then you need to go learn.  We need to find the path of 'mysqldump' with the 'which' command, create the backup directory, then add the following line after you get into 'crontab -e':

# which mysqldump
/usr/local/bin/mysqldump
# mkdir /var/sqlbackup
# crontab -e
0 3 * * * /usr/local/bin/mysqldump -hlocalhost -uintranetguy -pd0ntTe3ll intranet > /var/sqlbackup/intranet.sql

And every morning at 3am, you'll be backing up the intranet database.

4. Restoring databases.

So now you have to restore the database.  Super easy. From the host system shell:

$ mysql -u intranetguy -p intranet < /var/sqlbackup/intranet.sql

The system will prompt you for the intranetguy's password and the intranet database will be overwritten with the contents of the backup sql file.

Change in IP, reboot? Not with FreeBSD.

There are times when I need to swap a server replacement in for a machine that's gone down.  After changing the IP address in /etc/rc.conf there is a simple set of commands that can be strung together that will keep you from having to wait for a whole system reboot.

Step 1: change the IP address in /etc/rc.conf

ifconfig_re0="inet 10.10.10.49  netmask 255.255.255.0"

becomes

ifconfig_re0="inet 10.10.10.50  netmask 255.255.255.0"

As long as we didn't have to adjust the default router (which is just one other line in the same file) we can save the /etc/rc.conf file and issue these commands:

/etc/rc.d/netif restart && /etc/rc.d/routing restart

Your network will reset and start up with the new settings in the /etc/rc.conf file.  This will terminate any sessions currently open, including your SSH session, if that's how you're connected.

Fun with 'find'.

Here are some basic examples of the UNIX 'find' command that I use frequently. 


If you wanted to search for, and erase every file on your system named 'bahdouche' execute:

find / -name bahdouche -type f -exec rm -f {} \;

If you wanted to do a 'test run' before you actually killed them all, use the 'echo' command nested in the command line:

find / -name bahdouche -type f -exec echo rm -f {} \;

To find every file owned by the user named 'bahdouche' execute:

find / -user bahdouche -print

The -print flag is optional on some systems.

Find every file that has no user associated:

find / -nouser

Let's say you run a mail server with pam accounts and maildirs and you wanted to have a cron job that deletes their Spam folders every 30 days; add this to crontab:

find /home/*/Maildir/.Spam -type f -mtime +30 -exec rm -f {} \;

This obviously would have to be modified if the Spam folders were named differently.

Migrating user accounts in FreeBSD.

I make a lot of servers.  When it comes time for backups, upgrades, updates and general data security, I find myself moving users from one box to another quite frequently.  Typing in all of those usernames and passwords (that you probably have no way to know) might present a problem.  Luckily FreeBSD makes this process quite simple.  The first step (in anything computer-related) is to back up what you're about to touch.  This is as simple as these 3 commands on the server that is to host the new accounts:

cp /etc/master.passwd /etc/master.passwd.orig
cp /etc/passwd /etc/passwd.orig
cp /etc/group /etc/group.orig

Then copy these same files from the 'old' machine, you can use 'scp' or whatever to copy them.  If you're unsure how to do this, send me a message; that would be a whole separate post.  Once you get the files to the 'new' machine, copy them to the /etc folder overwriting what is there.  Type the following command to import all of the users:

pwd_mkdb /etc/master.passwd

(NOTE: THIS WILL MAKE THE USER LIST THE SAME AS THE 'OLD' MACHINE; ANY ACCOUNT THAT IS NOT ON THE OLD MACHINE WILL NOT EXIST; ROOT PASSWD WILL CHANGE TO WHAT 'OLD' IS SET TO.)

That is it. All of the users that could log into the 'old' server can now log onto the 'new' server with their same passwords.

FreeBSD and the sound card.

If you want to get your sound set up in FreeBSD, the handbook is an excellent place to start.  I highly suggest anyone using FreeBSD to familiarize yourself the handbook.  I wont repeat the steps on getting the card to work; after you do, if you still are not getting sound out of your speakers you might want to try some different mixer settings.

For whatever reason, it seems that some of the switches compete with each other not allowing for the sound to actually work.  What I did was a trial and error approach and landed on a setting that works for my hardware configuration.  Basically, you have to mute/unmute things that might or might not conflict with the accessories (mic, speakers, etc) you have plugged in.

First, type 'mixer' without any switches and see where you're at.  You will get a listing similar to the following output:

Mixer vol      is currently set to  78:78
Mixer pcm      is currently set to  75:75
Mixer speaker  is currently set to  75:75
Mixer line     is currently set to  75:75
Mixer mic      is currently set to   0:0
Mixer mix      is currently set to   0:0
Mixer rec      is currently set to  75:75
Mixer igain    is currently set to   0:0
Recording source: mic

I was able to get sound out by typing

mixer line 0:0

 or

mixer line 75:75

I kept some music playing while I changed settings for 'line' 'mic' 'pcm' 'speaker' , etc. until I found one that worked.  I'm sorry if this isn't the scientific answer you may have been searching for, but this is what worked for me on a few different machines.  Depending on your setup, what you mute/unmute might vary from my example.  What I landed on was this:

Mixer vol is currently set to 77:77
Mixer pcm is currently set to 75:75
Mixer speaker is currently set to 0:0
Mixer line is currently set to 75:75
Mixer mic is currently set to 75:75
Mixer mix is currently set to 0:0
Mixer rec is currently set to 0:0
Mixer igain is currently set to 0:0
Recording source: mic

Hope that saves someone the headache I had.

Hello world

class HelloWorldApp {
    public static void main(String[] args) {
        System.out.println("Hello World!"); // Display the string.
    }
}