Check your assumptions

I just went through one of those random Linux trial-by-fire exercises.  I had two web servers, one cloned from the other, behaving differently: One would send emails, the other wouldn’t.

After walking through the tree of all possible things that could be the problem: user installed software, system software, system configurations, and right down to logs and individual files…  and then realized the most obvious source of the difference:  The processes that handle mail on one server had died, but not on the other.

Yes, 2 hours of debugging mail handling on a linux machine only to discover that I could have figured this out with “ps aux | grep mail” in about 10 seconds, had I known what to look for.

Well, that’s the nature of troubleshooting – the answer is always obvious after you find it.

On the bright side, it means my lab has a shiny new blog to play with – and it seems like everything is working now.

 

Civilization V on Kubuntu 11.10

Just in case I forget, I managed to get Civilization V to run in Kubuntu 11.10 this evening.  It required the following:

sudo apt-get install wine
winetricks d3dx9 vcrun6 vcrun6sp6 vcrun2003 vcrun2005 vcrun2008 vcrun2010
winetricks corefonts flash
winetricks vcrun2008

Setting wine to work as Windows 2008, and then selecting direct x 9 (not direct x 11) did the trick.

I have some odd audio problem with it conking out after a few minutes, but otherwise, it ran reasonably well. Time to test it out for a few more minutes, while I get the puppy acclimatized to her kennel. (-:

How I recovered a SATA hard drive

Last night, I had the unpleasant experience of having a hard drive “crash” on me.  The computer turned on, but just sat there, with the ubuntu boot screen essentially stuck indefinitely.  The cause wasn’t too hard to figure out – one of my hard drives had died suddenly. Unfortunately, it was the hard drive to which I’d been diligently moving all my data, preparing the computer to be given away. (Anyone want a giant tower, two monitors and a four year old computer that supports up to 8 hard drives?)

In any case, since I was able to determine that the failed drive was SATA and not IDE, the solution turned out to be a reasonably scary, but simple solution.  (Btw, determining that the drive was broken consisted of unplugging the drive, and working out that the computer was able to boot without it, as it was a storage volume and did not have any system files on it.)

First, I allowed the computer to boot up to where it was stalled.  At this point, I unplugged the SATA cable (NOT the power cable) connecting the drive to the mother board.  The computer was then able to recognize that the boot process was stalled and was unable to mount the drive.  A prompt comes on the boot screen and asked if I wanted to manually repair (R) the drive or skip mounting (S) it.  Picking manual repair by pressing r, Ubuntu then drops you down to a shell prompt.

Once at the shell prompt, you can do something strange – plug the SATA cable back in.  For me, the drive was recognized, but error messages flashed up. (Alas, I didn’t write them down.)

At this point, I tried a manual mount on the drive – and it stalled.  20 minutes later, the drive was not mounted, which seemed unusual.  My suspicion was that the drive was being read but that the file system was misbehaving.  If the drive was truly dead, it should have failed much faster than that.

The solution then seemed somewhat intuitive: running “fsck -v /dev/<drivename>”.  Surprisingly, It was able to see the drive and began recovering the journaling.  After about 35 minutes, the drive was restored to a readable state, and all of my data appeared to be intact.

Not that there’s a moral to this story or anything, but I’m glad I didn’t give up on this disk as it turned out to be a file system problem, not a hardware problem.

However, there were two important things I did learn: 1) SATA hot swapping actually works on desktop computers. 2) If your EXT3/EXT4 file system fails, it may be indistinguishable from a hardware failure at first glance.  When in doubt, try a file system recovery – and SATA is pretty nifty!

(K)Ubuntu – Oneiric Ocelot is out… meh.

Ok, I wasn’t going to blog until I’ve finished my thesis corrections, but I’ve spent the whole day formatting protein/gene names correctly, and I’m showing signs of brain calcification.  So, I’m going to do one post that has nothing to do with science.  I’m saving those for later.

Instead, I thought I’d give up a few thoughts on Ubuntu’s latest release: Oneiric Ocelot (11.10).

First, I have a few caveats.  I’ve been using Onieric for several months.  It’s something I like to do: load up alpha versions and watch them develop.  It keeps you sharp: when you have to trouble shoot things, you learn lots about how the operating system works and you get the pleasure of finding new software improvements all the time.  What’s not to like? At this point, I’ve been running alpha and beta versions on my laptops for about 4 years now, and it has been an enjoyable process – well, at least it was until Natty (11.04).

For the past year, I’ve been increasingly disappointed in Ubuntu because more things get broken than fixed as the development goes along. But, I should explain a few things:  I don’t use the vanilla Ubuntu.

First, I use Kubuntu.  I’ve flipped-flopped between KDE and GNOME a few times, but I always find myself gravitating back to KDE.  The ability to customize things so deeply has always kept me coming back. (For instance, I remapped the Eject button on my macbook to the eject command last night in the KDE settings panel, and I think that’s the cat’s pajamas.  I’ve done the similar things to enable my keyboard backlight as well.  Stuff like that just makes me happy.)

Second, I love compiz.  I find KDE’s effects kinda lackluster, whereas compiz has a few modules that make me more productive.  I love scale and the desktop cube, because I think well in terms of a 3D desk, and it just makes it easy for me to remember where I left my windows.

So, with that in mind, My ideal desktop is Ubuntu + KDE + compiz.  A combination that ran REALLY well in 10.10.  Actually, that was the last time it did run well – which is a big part of my beef.

After several months of watching Oneiric evolve, I’m really disappointed. I was hoping for many more bug fixes before today’s release, but I just got more and more bugs.  Here are some of the ones that annoy me on a frequent basis:

  • Compiz no longer runs smoothly in KDE.  Flickering and artifacts that were all cleaned up in Maverick are all back with a vengeance.
  • Compiz has been broken so that it hangs repeatedly whenever KDE panels are used – particularly stuff like the new activity manager, or even just resizing a panel.
  • Compiz itself now prevents panels from appearing when it’s set as the default window manager, so that you must kill compiz and start the KDE window manager on every boot just to see your panels.
  • KDE itself has been broken so that the window manager crashes on EVERY single exit.  You can’t shut down the computer without having to hit close on two separate “KDE window manager has crashed” windows
  • There’s the dreaded “Kernel is taking too much power” bug that was only recently “fixed”.  Actually, it wasn’t fixed, but with a few custom kernel parameters, becomes manageable.
  • Oneiric decided to go with the brcmsmac driver for my wireless card.  The driver works fine, but it’s in developmental stage, so there’s no power management for the driver, making it take up 3-5W of power.  For a laptop, that’s just inexcusable – it’s about 30-50% of my total battery draw (usually 11-15W after a lot of manual tuning)!
  • Something totally botched up the NVidia dual monitor support. Even a month ago I was able to drive a second monitor from my video card, but it now fails to do so reliably.  I gave up the second monitor because turning the second monitor on and off 10-15 times in a row in the Nvidia settings panel just to get it to work is no longer a reasonable solution I’m willing to engage in.
  • Then there is the move to “alternate architecture” – I’m still not sure how this is supposed to work, but try getting skype to work.  It’s “fun”.  (Fun being defined as a pain in the ass that involves manually installing i386 libraries that are automagically removed every time you upgrade a package, because the 64-bit version just plain fails to work at all.)
  • Oh, and now Skype can’t see my microphone, but I’m not sure that’s an Ubuntu problem, although I doubt anything will be fixed now that Microsoft has bought Skype. (Again, not an Ubuntu problem…)

Anyhow, you get the idea.

It seems that KDE isn’t a priority for Ubuntu developers, and worse, I don’t think that Ubuntu devs are even aware of the breakage they’ve caused in compiz and KDE while re-purposing it for Unity.  Lack of testing might be one problem, but I suspect that they’re not really even interested in keeping compatibility – which was always one of the core virtues of GNU/Linux for me: interchangable parts.  I have no interest in switching to Unity, but I wish they wouldn’t break everything else for me in the rush to get Unity working for themselves.

Alas, while I’m going to keep using Kubuntu for a little while longer, my love for the Ubuntu distros is fading.  I love bleeding edge, but I’m not a fan of this rampant (avoidable) breakage.

So, my advice – stay away from Kubuntu Oneiric Ocelot – it’s not worth the pain.  With any luck, some of these bugs will be fixed for the LTS release Pulverized Penguin Precise Pangolin.  But I won’t be holding my breath.

Upgrading directly from Ubuntu Maverick to Oneiric Alpha 3.

First of all, this is my experience on macbook pro.  Let me begin by saying this is not recommended, my notes are practically non-existent (I’m just typing this from memory and the “history” command) and Oneiric Alpha three isn’t officially ready for production systems.  That is to say, DO NOT TRY THIS AT HOME!

The other issue that you must be aware of is the ability to put your computer in a non-bootable state very easily.  If you start this process, you must finish it.  If you turn off your computer part way through or before all of the dependencies are configured, you may find yourself with an inoperable computer, or with some subsystems (such as networking) in a completely unusable state – forcing a complete reinstall.

Ok, those disclaimers out of the way, it IS possible to go directly from Maverick (Ubuntu 10.10) to Oneiric (Ubuntu 11.11), but I don’t recommend it.  The process took hours (for me, 5 hours) of (learning how to) manually straighten out dependencies.  It was absolutely not worth the pain.  (It took approximately 500 install/remove/configure commands to straighten out, so consider yourself forewarned!)

To get it started, I brute forced the upgrade by opening my /etc/apt/sources.list file and doing a global search and replace, changing “maverick” to “oneiric”.  At that point, you can issue the command:

sudo apt-get update; sudo apt-get dist-upgrade

That will get the upgrade started.  If you’re ready for 5 hours of agony, press Y.  Before your trial by fire can begin, you may have to wait about while the nearly 1.3Gb of updated packages download. It took about 2 hours for me.

At first, the process will look fine, installing packages quickly, with the update manager doing what it should. However, at some point, the whole upgrade crapped out on me with unresolvable dependencies.  At this point, I used two tools:

sudo apt-get dist-upgrade

and

sudo dpkg --install [deb filename]
sudo dpkg --remove [package name]

Between those two tools (and three commands), you can work through all but one of the broken package problems you’ll face. (I used synaptic to bypass that one, as mentioned below.)

Unfortunately, I don’t have any screen captures, but if you try, you’ll see a message that reads something like:

Package [name1] can't be installed, depends: [name2] (>version).
  [name2] not configured.

Each time, you’ll need to manually try to add package [name2] yourself.  The trick is that all of these files are already downloaded, so you can add them relatively easily by moving into the directory in which apt downloaded them:

cd /var/cache/apt/archives/

The (not so hard) challenge is to figure out the name of the .deb file you need to install – which is surprisingly easy.  Once you’re in the apt archives directory, all of the packages will start with the name of the package as described by the error message above.

The bigger problem is that the dependency relations can be pretty deep.  Sometimes trying to install one package will result in another message that it, in turn, depends on another package.  One by one, you have to follow each lead to figure out what the right package is, and worse, the right order to install them in.

The bigger issue is that periodically, you will find that there is a set of dependencies that you just can’t resolve, in which case, you need to remove the blocking package from the system, install the new package, then re-install the new version of the one you’ve just un-installed.  For instance, I found I had to remove the kubuntu-desktop package to get some of the kdebase packages to install.  It’s not a big deal to do this (although you need to be aware you can accidentally remove a lot of important stuff if you aren’t careful), but you’ll need to be aware of what you remove so you can add it back if (and only if!) there is an updated package for it in the archives directory.

I also found, at one point, that there were packages I could not resolve (icedtea-6-jre-jamvm and openjdk-6-jre, if I remember right) and ended up turning to synaptic. There was a simple “resolve broken packages” command or some such. That was able to get around the set I couldn’t fix manually. I still have no idea how. However, when Synaptic works, it can finish an install for you – while it can’t/doesn’t fix everything, it can be an incredibly useful tool for navigating some complex dependency nightmares.

Periodically, you’ll also discover that you have a bunch of things that are installed, but not configured because their dependencies weren’t satisfied.  You can either re-install them manually, or use the command:

sudo dpkg --configure -a

that will reconfigure any packages in that state.

Other notes – I had to remove xserver-xorg-video-all and xserver-xorg-video-nouveau to get X to work again, as well as blacklist the nouveau driver.  You can see my post here, where I describe it in more detail.  I also had to reinstall kubuntu-desktop at the end, as it was removed earlier.

Finally, I also found I needed to recompile my kernel modules (eg, for nvida-current)at some point when I did a further dist-upgrade.  For those who are familiar with the process, I discovered this on a later reboot when x refused to restart and had to drop to the terminal to make that change.

To give the complete picture, even now, not everything is now working.  Adjusting KDE plasma panels can cause significant instability in compiz (driving it up to 100% cpu), and I had to reset the compiz settings to default before compiz would stop becoming unstable. (I have a diff between the before and after settings, and nothing jumps out at me other than the active plugins and the s0_outputs resolution:

< s0_active_plugins = core;bailer;detection;composite;opengl;imgjpeg;
            debugspew;compiztoolbox;decor;vpswitch;mousepoll;resize;
            commands;imgpng;imgsvg;staticswitcher;move;dbus;resizeinfo;
            regex;text;wobbly;place;session;animation;kdecompat;fade;
            cube;rotate;expo;scale;scaleaddon;
---
> s0_active_plugins = core;bailer;composite;opengl;debugspew;decor;
            cube;kdecompat;vpswitch;resize;imgpng;rotate;move;scale;
            snap;place;

and

< s0_outputs = 1680x1050+0+0; 
--- 
> s0_outputs = 640x480+0+0;

I can’t see that helping anyone, but perhaps one of the packages previously enabled was the cause of the instability.

Hopefully the above is useful to someone else, either by pointing you in the right directions if you follow my footsteps, OR, by keeping someone from following. Either one would be a good outcome!

Good luck, and happy migrating.

Compiz running on Ubuntu 11.10 (Oneiric)

NOTE: October 18, 2011.  I see that this page is the top hit on my blog this week.  I think if you’re looking for more up to date information, you should see my current opinion on Compiz on Kubuntu here.

After months of waiting for things to be resolved with the nvidia drivers and such, I finally buckled down to see if I could get compiz running again smoothly on Ubuntu 11.10.  Happily, I can report that the answer is yes!

Using the latest packages, the trick was to follow some advice in this thread.

First, you need to blacklist the nouveau driver.  It hasn’t been working for me, and it has been preventing the nvidia driver from being loaded.

sudo nano /etc/modprobe.d/blacklist.conf

add “blacklist nouveau” to the end of the file, then save and close it.

Next, you do need to update initramfs and  reinstall the nvidia-current driver:

sudo update-initramfs -u -v
sudo apt-get install --reinstall nvidia-current

From that point on, a simple reboot (and making sure that your xorg.conf actually uses the nvidia driver) should be enough to get you back into Xorg with the Nvidia binary blob… and by opening a terminal and typing:

compiz --replace

You should be good to go again. Things appear to be stable, but I’m sure this will need a lot more testing before I switch back to compiz/kde/nvidia as my default setup.

Random OpenOffice 3.2 tip

Just in case I ever come across this problem again.

If OpenOffice 3.2 repeatedly crashes on dialog boxes for open/save/insert image/etc, the fix is to install a single package that seems to import the proper dialog box tool kit: openoffice.org-gtk

This is probably something that only happens to Kubuntu users, as openoffice.org-gtk doesn’t sound like a package you’d need on kde.

Anyhow, if you’re using kubuntu, the fix is as simple as:

sudo apt-get install openoffice.org-gtk

And instantaneously, everything works again.

Fixing the screen/LCD brightness keys on a macbook pro, Ubuntu 10.10.

If you’ve been following my posts, you’ll know I have a macbook pro running Ubuntu Linux, and you’ll also know I love tweaking things. I’m not obsessive about tweaking, but if something could work better, I’d like it to work better. So after a month of having 4 dead keys on my laptop, I figured I had to do something about it.

They keys are:

  • keyboard back light brightness up
  • keyboard back light brightness down
  • monitor brightness up
  • monitor brightness down

They’re hardly the most important buttons on a keyboard, but I figured I’d let them sit idle long enough.

Getting them to work turns out to be a relatively simple.  Initially, I’d just followed the instructions here to get the keyboard LED brightness working, and that did a decent job…. but not perfect.  It didn’t actually let you get the brightness all the way to zero or to 255, the max brightness.   Thus, I modified the script:

(You can download it here)

#!/bin/bash
# Francisco Diéguez Souto (frandieguez@ubuntu.com)
# This script is licensed under MIT License.
# Modified by Anthony Fejes (apfejes@gmail.com)
#
# This program just modifies the value of backlight keyboard for Apple Laptops
# You must run it as root user or via sudo.
# As a shortcut you could allow to admin users to run via sudo without password
# prompt. To do this you must add sudoers file the next contents:
#
#   ALL = (ALL) NOPASSWD: /usr/sbin/keyboard-backlight
#
# You must then install the script in the path given above, eg, /usr/sbin/
# If you chose another path, then the location in the sudoers file must reflect 
# that path.
#
# After this you can use this script as follows:
#
#     Increase backlight keyboard:
#           $ sudo keyboard-backlight up
#     Decrease backlight keyboard:
#           $ sudo keyboard-backlight down
#
# You can customize the amount of backlight by step by changing the INCREMENT
# variable as you want it.

BACKLIGHT=$(cat /sys/class/leds/smc::kbd_backlight/brightness)
INCREMENT=10

if [ $UID -ne 0 ]; then
    echo "Please run this program as superuser"
    exit 1
fi                                                                                                     
                                                                                                       
SET_VALUE=0                                                                                            
                                                                                                       
case $1 in                                                                                             
                                                                                                       
    up)                                                                                                
        TOTAL=`expr $BACKLIGHT + $INCREMENT`
        if [ $BACKLIGHT -eq "255" ]; then
                exit 1
        fi
        if [ $TOTAL -gt "255" ]; then
            TOTAL="255"
        fi
        echo $TOTAL > /sys/class/leds/smc::kbd_backlight/brightness
        ;;
    down)
        TOTAL=`expr $BACKLIGHT - $INCREMENT`
        if [ $BACKLIGHT -eq "0" ]; then
                exit 1
        fi
        if [ $TOTAL -lt "0" ]; then
            TOTAL="0"
        fi
        echo $TOTAL > /sys/class/leds/smc::kbd_backlight/brightness
        ;;
    *)
        echo "Use: keyboard-light up|down"
        ;;
esac

Following the instructions in the header to add this into the sudoers file, you can then go to your windowing environment and associate the keys with the command. In my case, I went to the Settings menu in the KDE launcher, clicked on the System Settings toolbox and went into the Gestures and Shortcuts menu. I created a new group brightness controls in the custom input action settings, then used the menu to create new global shortcuts and picked the “command/URL” type. At this point, all you need to do is move to the “Trigger” tab, click on the key you want to associate with each command, then enter the commands into the action tab. (The commands are:

sudo /usr/sbin/keyboard-backlight up

and

sudo /usr/sbin/keyboard-backlight down

Getting the screen brightness to work wasn’t that much harder. The script looks like:

(You can download it here)

#!/bin/bash                                                                                            
                                                                                                       
# Anthony Fejes (apfejes@gmail.com)                                                                    
# Template taken from post by Fran Diéguez at                                                          
# http://www.mabishu.com/blog/2010/06/24/macbook-pro-keyboard-backlight-keys-on-ubuntu-gnulinux/       
#                                                                                                      
# This program just modifies the value of video brightness for Apple Laptops                           
# You must run it as root user or via sudo.
# As a shortcut you could allow to admin users to run via sudo without password
# prompt. To do this you must add sudoers file the next contents:
#
#   ALL = NOPASSWD: /usr/sbin/mbp_backlight

# After this you can use this script as follows:
#
#     Increase backlight keyboard:
#           $ sudo mbp_backlight up
#     Decrease backlight keyboard:
#           $ sudo mbp_backlight down
#

BACKLIGHT=$(cat /sys/devices/virtual/backlight/mbp_backlight/brightness)
MAX=$(cat /sys/devices/virtual/backlight/mbp_backlight/max_brightness)
MIN=4
INCREMENT=1

if [ $UID -ne 0 ]; then
    echo "Please run this program as superuser"
    exit 1
fi

case $1 in

    up)
        TOTAL=`expr $BACKLIGHT + $INCREMENT`
        if [ $BACKLIGHT -eq $MAX ]; then
                exit 1
        fi
        if [ $TOTAL -gt $MAX ]; then
            let TOTAL=MAX
        fi
        echo $TOTAL > /sys/devices/virtual/backlight/mbp_backlight/brightness
        ;;
    down)
        TOTAL=`expr $BACKLIGHT - $INCREMENT`
        if [ $BACKLIGHT -eq $MIN ]; then
                exit 1
        fi
        if [ $TOTAL -lt $MIN ]; then
            let TOTAL=MIN
        fi
        echo $TOTAL > /sys/devices/virtual/backlight/mbp_backlight/brightness
        ;;
    *)
        echo "Use: mbp_backlight up|down"
        ;;
esac

And method is identical to that above, associating the appropriate keys to the command:

sudo /usr/sbin/mbp_brightness up

and

sudo /usr/sbin/mbp_brightness down

And I’m now happily able to use all of the keys on my keyboard!