This is why I use unix-like...

Here's the story.  We build a server for biometric authentication, right?  Piece of cake.  Except for the fact that the USB devices only work in an OS that starts with Win, but ends with "Doh!"s.

It gets carried out to the data center and racked.  Wont come online; no ping, and no RDP present.  Network Engineer gets it online with the IP KVM and manages to get it online.  Windows Firewall was blocking the traffic.

What?  No, that can't be possible, because when we shut it down the last two things that happened was to change to the DC IP and turn off Windows Firewall.

So I get the software installed and the appserver connected to it; since I'm just the engineer, I don't have any IDs or anything to test with so I call it a night.  The next morning the technicians test it out and everything is working- yay.

That night I'm on the VPN and brushing up the second instance of the biometric server, or at least I was trying to, but I couldn't get to the server.  I figured there was something up with the VPN again and it lost a route or something.  The next morning at work I still can't get on it.  So the network guy jumps on the IP KVM and log on.  Windows Firewall had turned itself back on, again.  This machine is off-domain with a pretty minimal config and everything was turned off with the Administrator user.

So when I get RDP back, I turn off Windows Firewall, again.  Then I turn off the service Windows Firewall.  There; try to get out of that you ... bad server.

Because I'm OCD, I try to work from home again that night.  This time ... no, I still can't get on it, but it has to be the VPN this time.  I get to work the next day, today, which is exactly one day before this biometric authentication server is to go live for tens of thousands of people, and it's offline again.

I know what you're thinking; "this can't be happening; it's made up."

I can't make this up.

Windows Firewall wasn't on, though.  Sometime in the night it did Windows Updates and rebooted itself.  A server decided to reboot itself because it wanted to.  This is OK, right?  Who cares what the server might be serving; heck, there was an update to .Net!

I'm not using any .Net services.

None of that was the real problem, though.  The real problem is the static IP that was configured and working the day before was suddenly 169.254.... Yes, an autoconfig IP address when a DHCP server isn't available.

It's not DHCP, it's static.  It's not on the domain; it's stand-alone.  Seriously what is happening?

TCP/IP is damaged, to make a long story shorter.  Why?  Who cares.  How do I fix it?

Like this:

netsh int ip reset c:\resetlog.txt

shutdown -r -t 00

The first command will reset the TCP/IP.  Be aware, that the second command is going to do exactly what it says; you're going to shutdown and reboot without waiting at all.

When it comes back up you will have to set your static IP again.
Then hopefully your ping -t <hostname> will start returning.

Stupid.

LoReM iPsUM awk

Did you ever get a list of stuff, like email addresses or domain names or something, and the person who gave it to you did crazy capitalization?  Some are all lowercase, some are uppercase, but the only thing that's consistent is there's no commonality.

awk to the rescue again.

Let's assume you have a list of stuff you've copied and pasted into a text file called delete.txt.  This one-liner will amend your ailment:

 cat delete.txt |awk '{print tolower ($0)}'

Voila.

That's what she sed.

More sed magick today in the salt mines.

Here was my issue.  I was moving dozens of domains from their position on a cluster server, and moving over to another machine that was to take over as the cluster 'master'.  Since the server they were on was a live server, I couldn't just move the IP addresses without blowing up all of those sites, and DNS would tell me that the old server was working fine; how am I going to test the sites on the new server before I flip the switch?

Well, edit the hosts file on my local machine of course.  Hrm.  There might be 100 sites that I'd need to change the IP for in my hosts file; that's going to suck.

Not really, because my friends sed and awk can make quick work of this.

Assuming:

A) I have 2 servers that have multiple vhost entries in a file called client-vhost.conf
B) the server IP addresses bound to apache are x.x.x.x (server1) and y.y.y.y (server2)
C) the public document root for the websites have subfolders that represent the domain name:

(ie. drwxrwxr-x 7 www www 4096 Dec 1 2011 example.com)

D) we've already synced the data and config files so they are identical on both machines.
E) 5.X CentOS install with configs in /etc/httpd/conf/ and data folder of /var/www/

Since server1 has been up and running serving web sites for a period of time, all of our work is going to be on server2.  Edit the client-vhost.conf file, you're going to need to do the first part by hand.  Change the IP apache is listening on.  We don't want to pull out the old entries, because server2 is going to take the IPs from server1. Comment out the x.x.x.x lines and insert the y.y.y.y lines:

#Listen x.x.x.x:80
#NameVirtualHost x.x.x.x
Listen y.y.y.y:80
NameVirtualHost y.y.y.y

Then save.

Now we're going to edit all of the vhost entries, there's a lot- so sed will do all the lifting for us.  From the command-line type:

sed -i 's/x.x.x.x/x.x.x.x\ y.y.y.y/g' /etc/httpd/conf/client-vhost.conf

That will replace the search string 's/----- with the global replace string /--------/g'.  We put a space in between the server1 IP address and the server2 IP address, so we need to escape the space with a '\'.

Now we need to get the new IP into my hosts file with all the domain names.  How can I get the columns displayed easily with the new IP and a space followed by the domain name?  Like this:

ll /var/www/ | grep com | awk '{print "y.y.y.y " $9}'

What I did there- 'll' is kinda like 'ls -la' and displays 9 columns; I'm greping 'com' for the *.com reference because there are potentially folders in there that aren't published domains, and I don't care about those; awk takes the piped information and prints "y.y.y.y " (<- with a space before the last ' " ') then $9 or the ninth column, which is the directory name.  This gave me a nice two-column list I can copy/paste into my c:\windows\system32\drivers\etc\hosts file (right-click notepad and Open as Administrator if you're using Windows post-XP) that looks like this:

y.y.y.y example1.com
y.y.y.y example2.com
y.y.y.y example3.com

Now I can just paste the domains into my browser and make sure they work on the new box before making any potentially destructive changes.  Which was a good call, because the real-life example this post was based off of had a few things missing in the server configuration.

iptables, oh no you didn't..

Being a systems administrator for a large number of linux systems that are primarily web and email servers, I get to see a lot of attacks on different services.

One of our boxes (A) was brought down from another box for the same client (B).  A. was running httpd and B. was running sendmail.  After examining the logs and determining that B. was the culprit, an examination of the logs on B. showed a particular IP with a large number of simultaneous connections.  His vector was to do dictionary/name email blast to the domain, then when he got a valid user, he would try to authenticate via another dictionary attack.  This was spawning a large number of sendmail processes that ended up slowing down the box considerably.

Performing a whois on the IP address showed us that the hacker was operating from another country that most likely was not a customer of our client.  

whois X.X.X.X

In order to keep the box and network alive, we decided to block the IP address on the machine's firewall.  We didn't want to add the IP to be there on reboot, in case it was a DHCP address that might be legitimate in the future, but we can insert the IP address inline, which will block it until the iptables is restarted.  If this IP comes through again in the future, we would add it to the /etc/sysconfig/iptables and reload the configuration.

This one-liner will keep any traffic from the 'bad' address from doing anything further on this box:

iptables -I INPUT -s X.X.X.X -j DROP

As soon as you hit enter, that attack is over.  If you realize you typed in the wrong IP address, and you want to remove the inline rule you just created, change the -I to a -D like this:

iptables -D INPUT -s 208.122.195.120 -j DROP

If you wanted to block this IP permanently, simply add this line to the /etc/sysconfig/iptables:

-A INPUT -s X.X.X.X -j DROP

Then reload your iptables config.

I BASHed the terminal and sed 's/ex...'

Like MySQL, BASH scripting is something I've always done frequently, but very light as far as complexity and depth.

Batching commands is easy, but now I'm coming into a new phase of my career where scripting is crucial to the day-to-day operations.  I'm finding help in the form of co-workers and tldp.org.  The following is a sanitized version of the check and insert I was asked to create that will turn on fasttcp on CentOS/RHEL, but not if the kernel is "2.6.18-238.9.1.el5.fsl_2.0.2.1"

So we need to:
a) Check kernel to exclude a certain build.
      i. (if yes) Insert 'kernel*' to the exclude on yum.conf
     ii. Report if the yum.conf was modified or not.
b) download and install
c) chkconfig on resources

I'm a total noob at scripting, so I made sure I didn't break anything on the system; I created the script as a non-root user and copied the yum.conf to that user's home folder with a unique name.

cp /etc/yum.conf /home/dave/dave-practice-yum.conf

First, the beginning of any (good) shell script calls the shell you're working in.  I'm from BSD most-recently, so !/bin/sh is what I would generally use, but now that I'm a CentOS guy at my new job, !/bin/bash is the shell for me.

#!/bin/bash

Defining variables: I need to find out if the kernel is the mystical kernel that isn't supposed to get updated, so I'll establish the KRNL variable.  I also am going to insert 'kernel*' into the 'exclude=' line of the yum.conf if it is a match, so I will also define the EXCL variable to make sure I'm not over-writing an already-established 'exclude='.

KRNL=`uname -r`
EXCL=`grep "exclude=" dave-practice-yum.conf|cut -d= -f2`

Now we write in the actual check with an IF statement:

if [ $KRNL = 2.6.18-238.9.1.el5.fsl_2.0.2.1 ]
then
        if [ -z $EXCL ]
        then
                sed -i 's/exclude=/exclude=kernel*/' dave-practice-yum.conf
                echo "yum.conf modified."
        else
                sed -i '/exclude=/s/$/,kernel*/g' dave-practice-yum.conf
        fi
else
        echo "No modification to yum.conf"
fi

The else in the nested condition is pretty ugly.  I didn't want to over-write if there were already any excludes, so I appended the 'kernel*' to the 'exclude=' if nothing were present; if there is something present on the exclude line when the script runs, it will simply put a ',kernel*'.  This works, but if 'kernel*' is there, it will be repeated.

Then we download the source package from the internal repository, and chkconfig the services on (in this case FASTTCP).

/usr/sbin/wget -qO - http://server/files/fasttcp/ftcp_rc.tar|tar xf - -C /
/sbin/chkconfig --add fasttcp
/sbin/chkconfig --levels 2345 fasttcp on
/etc/init.d/fasttcp start

Remarkably, this script works; thanks in large part to my co-workers, and James in particular.  Below, find the script in its entirety (without blogspot formatting).

#!/bin/bash

KRNL=`uname -r`

EXCL=`grep "exclude=" dave-practice-yum.conf|cut -d= -f2`

if [ $KRNL = 2.6.18-238.9.1.el5.fsl_2.0.2.1 ]

then

        if [ -z $EXCL ]

        then

                sed -i 's/exclude=/exclude=kernel*/' dave-practice-yum.conf

                echo "yum.conf modified."

        else

                sed -i '/exclude=/s/$/,kernel*/g' dave-practice-yum.conf

        fi

else echo "No modification to yum.conf"

fi

/usr/sbin/wget -qO - http://server/files/fasttcp/ftcp_rc.tar|tar xf - -C /

/sbin/chkconfig --add fasttcp

/sbin/chkconfig --levels 2345 fasttcp on

/etc/init.d/fasttcp start

A funny thing happened on the way to the Internet...

Has anybody else noticed the kooky stuff that's been going on pretty much globally?  We have experienced crazy DNS disappearances, terrible latency, dropped VPN connections, web servers unresponsive, and that's just the beginning.  I'm not just talking about the film studio, or my home-based business; I'm also talking about things like GoDaddy and Network Solutions (ok, I know, Network Solutions was hardly ever 'fast' to begin with).  Something Big and most certainly Bad is going on.

Terrible things are going on and I'm concerned that we are about to find out what.  Is it a DDoS at our backbones?  Are they poisoning DNS?  Who are they?  Will someone make me a sandwich?

Format a USB disk with ext3 in Red Hat style systems.

We have some Red Hat boxes that do environmental effects on some of the motion pictures we do.  These guys need to archive some of their stuff onto USB disks so they can transport them or give them to clients, etc.  The problem is that they usually come formatted with FAT32 or something, and the files can be greatly in excess of 2.0GB.  So we put ext3 on.

First, see if the USB disk mounted itself.  Sometimes Red Hat does this.  Use df or mount to see if it's there.
[root@discreet ]# df

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda3            151668928  43941540 100023016  31% /
/dev/sda1               101086     13285     82582  14% /boot
none                   3928520        16   3928504   1% /dev/shm
/dev/sdb1            721075720     70584 684376536   1% /mnt/usbdisk1

Unmount /mnt/usbdisk1.
[root@discreet ]# umount /mnt/usbdisk1
then fdisk it (must be root!)
 [root@discreet ]# /sbin/fdisk /dev/sdb

The number of cylinders for this disk is set to 91201.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Input a p to list current partitions.
Command (m for help): p

Disk /dev/sdb: 750.1 GB, 750156374016 bytes
255 heads, 63 sectors/track, 91201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1       91201   732572001    c  W95 FAT32 (LBA)  ### FAT DOES NOT ALLOW >2GB FILES

Delete the FAT32 partition with d.
Command (m for help): d
Selected partition 1


Do another p to display the partition list after you have deleted partition 1.
Command (m for help): p       

Disk /dev/sdb: 750.1 GB, 750156374016 bytes
255 heads, 63 sectors/track, 91201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System

We're going to hit n for new partition.
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p           ###### P FOR PRIMARY PARTITION


Partition number (1-4): 1
First cylinder (1-91201, default 1):
Using default value 1       ##### PRESS ENTER FOR CYLINDER 1 TO BE START OF PARTITION
Last cylinder or +size or +sizeM or +sizeK (1-91201, default 91201):   #### JUST PRESS ENTER FOR DEFAULT PARTITION TO TAKE UP WHOLE DRIVE
Using default value 91201

Print the partition table one more time to make sure it's ext3
Command (m for help): p      

Disk /dev/sdb: 750.1 GB, 750156374016 bytes
255 heads, 63 sectors/track, 91201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1       91201   732572001   83  Linux  

Write and quit with wq.
Command (m for help): wq               ####### W FOR WRITE, Q FOR QUIT
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Make the new file system.

[root@discreet ]# /sbin/mkfs /dev/sdb1

mke2fs 1.35 (28-Feb-2004)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
91586560 inodes, 183143000 blocks
9157150 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
5590 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000
Writing inode tables: done
Writing superblocks and filesystem accounting information: done

Mount your new partition where you like and copy large files to it.