Check your assumptions

I just went through one of those random Linux trial-by-fire exercises.  I had two web servers, one cloned from the other, behaving differently: One would send emails, the other wouldn’t.

After walking through the tree of all possible things that could be the problem: user installed software, system software, system configurations, and right down to logs and individual files…  and then realized the most obvious source of the difference:  The processes that handle mail on one server had died, but not on the other.

Yes, 2 hours of debugging mail handling on a linux machine only to discover that I could have figured this out with “ps aux | grep mail” in about 10 seconds, had I known what to look for.

Well, that’s the nature of troubleshooting – the answer is always obvious after you find it.

On the bright side, it means my lab has a shiny new blog to play with – and it seems like everything is working now.

 

Compiz on KDE4.8 – a few details

Just to follow up on my last post, apparently compiz on KDE 4.8 runs as long as you don’t use the KDE/QT Event Loop plugin, but do select the text plugin (CCSM, under image loading).

I don’t know the source of the instability, but if the KDE/QT plugin is selected, the following error is presented:

glibmm-ERROR **: 
unhandled exception (type std::exception) in signal handler:
what: call to empty boost::function

Trace/breakpoint trap (core dumped)

Also worth noting, I had trouble using the default window manager. But, if you place:

/usr/bin/kde4-window-decorator --replace

as the window manager (Compiz Settings Manager -> Window Decoration Plugin -> General Tab -> Command), it seems to work.

I’ll have to explore more, but it does seem to be working, for now.

Compiz works with KDE 4.8!

After waiting what felt like forever, compiz now works again for me with KDE 4.8. (That is to say, it works on Kubuntu 12.04 pangolin current build.)  At least, one of the major bugs (this is the one I filed, after complaining about it here) is fixed, which makes the compiz/KDE combination usable again.  I haven’t tested if some of the panels still do bizarre things like become permanent overlays on the screen, but I’ll get to those later.

In any case, I’m happy enough to be able to use KDE and compiz together again.  You know, I think I’m starting to like Pangolin… and it’s still on the first alpha!

Civilization V on Kubuntu 11.10

Just in case I forget, I managed to get Civilization V to run in Kubuntu 11.10 this evening.  It required the following:

sudo apt-get install wine
winetricks d3dx9 vcrun6 vcrun6sp6 vcrun2003 vcrun2005 vcrun2008 vcrun2010
winetricks corefonts flash
winetricks vcrun2008

Setting wine to work as Windows 2008, and then selecting direct x 9 (not direct x 11) did the trick.

I have some odd audio problem with it conking out after a few minutes, but otherwise, it ran reasonably well. Time to test it out for a few more minutes, while I get the puppy acclimatized to her kennel. (-:

How I recovered a SATA hard drive

Last night, I had the unpleasant experience of having a hard drive “crash” on me.  The computer turned on, but just sat there, with the ubuntu boot screen essentially stuck indefinitely.  The cause wasn’t too hard to figure out – one of my hard drives had died suddenly. Unfortunately, it was the hard drive to which I’d been diligently moving all my data, preparing the computer to be given away. (Anyone want a giant tower, two monitors and a four year old computer that supports up to 8 hard drives?)

In any case, since I was able to determine that the failed drive was SATA and not IDE, the solution turned out to be a reasonably scary, but simple solution.  (Btw, determining that the drive was broken consisted of unplugging the drive, and working out that the computer was able to boot without it, as it was a storage volume and did not have any system files on it.)

First, I allowed the computer to boot up to where it was stalled.  At this point, I unplugged the SATA cable (NOT the power cable) connecting the drive to the mother board.  The computer was then able to recognize that the boot process was stalled and was unable to mount the drive.  A prompt comes on the boot screen and asked if I wanted to manually repair (R) the drive or skip mounting (S) it.  Picking manual repair by pressing r, Ubuntu then drops you down to a shell prompt.

Once at the shell prompt, you can do something strange – plug the SATA cable back in.  For me, the drive was recognized, but error messages flashed up. (Alas, I didn’t write them down.)

At this point, I tried a manual mount on the drive – and it stalled.  20 minutes later, the drive was not mounted, which seemed unusual.  My suspicion was that the drive was being read but that the file system was misbehaving.  If the drive was truly dead, it should have failed much faster than that.

The solution then seemed somewhat intuitive: running “fsck -v /dev/<drivename>”.  Surprisingly, It was able to see the drive and began recovering the journaling.  After about 35 minutes, the drive was restored to a readable state, and all of my data appeared to be intact.

Not that there’s a moral to this story or anything, but I’m glad I didn’t give up on this disk as it turned out to be a file system problem, not a hardware problem.

However, there were two important things I did learn: 1) SATA hot swapping actually works on desktop computers. 2) If your EXT3/EXT4 file system fails, it may be indistinguishable from a hardware failure at first glance.  When in doubt, try a file system recovery – and SATA is pretty nifty!

(K)Ubuntu – Oneiric Ocelot is out… meh.

Ok, I wasn’t going to blog until I’ve finished my thesis corrections, but I’ve spent the whole day formatting protein/gene names correctly, and I’m showing signs of brain calcification.  So, I’m going to do one post that has nothing to do with science.  I’m saving those for later.

Instead, I thought I’d give up a few thoughts on Ubuntu’s latest release: Oneiric Ocelot (11.10).

First, I have a few caveats.  I’ve been using Onieric for several months.  It’s something I like to do: load up alpha versions and watch them develop.  It keeps you sharp: when you have to trouble shoot things, you learn lots about how the operating system works and you get the pleasure of finding new software improvements all the time.  What’s not to like? At this point, I’ve been running alpha and beta versions on my laptops for about 4 years now, and it has been an enjoyable process – well, at least it was until Natty (11.04).

For the past year, I’ve been increasingly disappointed in Ubuntu because more things get broken than fixed as the development goes along. But, I should explain a few things:  I don’t use the vanilla Ubuntu.

First, I use Kubuntu.  I’ve flipped-flopped between KDE and GNOME a few times, but I always find myself gravitating back to KDE.  The ability to customize things so deeply has always kept me coming back. (For instance, I remapped the Eject button on my macbook to the eject command last night in the KDE settings panel, and I think that’s the cat’s pajamas.  I’ve done the similar things to enable my keyboard backlight as well.  Stuff like that just makes me happy.)

Second, I love compiz.  I find KDE’s effects kinda lackluster, whereas compiz has a few modules that make me more productive.  I love scale and the desktop cube, because I think well in terms of a 3D desk, and it just makes it easy for me to remember where I left my windows.

So, with that in mind, My ideal desktop is Ubuntu + KDE + compiz.  A combination that ran REALLY well in 10.10.  Actually, that was the last time it did run well – which is a big part of my beef.

After several months of watching Oneiric evolve, I’m really disappointed. I was hoping for many more bug fixes before today’s release, but I just got more and more bugs.  Here are some of the ones that annoy me on a frequent basis:

  • Compiz no longer runs smoothly in KDE.  Flickering and artifacts that were all cleaned up in Maverick are all back with a vengeance.
  • Compiz has been broken so that it hangs repeatedly whenever KDE panels are used – particularly stuff like the new activity manager, or even just resizing a panel.
  • Compiz itself now prevents panels from appearing when it’s set as the default window manager, so that you must kill compiz and start the KDE window manager on every boot just to see your panels.
  • KDE itself has been broken so that the window manager crashes on EVERY single exit.  You can’t shut down the computer without having to hit close on two separate “KDE window manager has crashed” windows
  • There’s the dreaded “Kernel is taking too much power” bug that was only recently “fixed”.  Actually, it wasn’t fixed, but with a few custom kernel parameters, becomes manageable.
  • Oneiric decided to go with the brcmsmac driver for my wireless card.  The driver works fine, but it’s in developmental stage, so there’s no power management for the driver, making it take up 3-5W of power.  For a laptop, that’s just inexcusable – it’s about 30-50% of my total battery draw (usually 11-15W after a lot of manual tuning)!
  • Something totally botched up the NVidia dual monitor support. Even a month ago I was able to drive a second monitor from my video card, but it now fails to do so reliably.  I gave up the second monitor because turning the second monitor on and off 10-15 times in a row in the Nvidia settings panel just to get it to work is no longer a reasonable solution I’m willing to engage in.
  • Then there is the move to “alternate architecture” – I’m still not sure how this is supposed to work, but try getting skype to work.  It’s “fun”.  (Fun being defined as a pain in the ass that involves manually installing i386 libraries that are automagically removed every time you upgrade a package, because the 64-bit version just plain fails to work at all.)
  • Oh, and now Skype can’t see my microphone, but I’m not sure that’s an Ubuntu problem, although I doubt anything will be fixed now that Microsoft has bought Skype. (Again, not an Ubuntu problem…)

Anyhow, you get the idea.

It seems that KDE isn’t a priority for Ubuntu developers, and worse, I don’t think that Ubuntu devs are even aware of the breakage they’ve caused in compiz and KDE while re-purposing it for Unity.  Lack of testing might be one problem, but I suspect that they’re not really even interested in keeping compatibility – which was always one of the core virtues of GNU/Linux for me: interchangable parts.  I have no interest in switching to Unity, but I wish they wouldn’t break everything else for me in the rush to get Unity working for themselves.

Alas, while I’m going to keep using Kubuntu for a little while longer, my love for the Ubuntu distros is fading.  I love bleeding edge, but I’m not a fan of this rampant (avoidable) breakage.

So, my advice – stay away from Kubuntu Oneiric Ocelot – it’s not worth the pain.  With any luck, some of these bugs will be fixed for the LTS release Pulverized Penguin Precise Pangolin.  But I won’t be holding my breath.

Upgrading directly from Ubuntu Maverick to Oneiric Alpha 3.

First of all, this is my experience on macbook pro.  Let me begin by saying this is not recommended, my notes are practically non-existent (I’m just typing this from memory and the “history” command) and Oneiric Alpha three isn’t officially ready for production systems.  That is to say, DO NOT TRY THIS AT HOME!

The other issue that you must be aware of is the ability to put your computer in a non-bootable state very easily.  If you start this process, you must finish it.  If you turn off your computer part way through or before all of the dependencies are configured, you may find yourself with an inoperable computer, or with some subsystems (such as networking) in a completely unusable state – forcing a complete reinstall.

Ok, those disclaimers out of the way, it IS possible to go directly from Maverick (Ubuntu 10.10) to Oneiric (Ubuntu 11.11), but I don’t recommend it.  The process took hours (for me, 5 hours) of (learning how to) manually straighten out dependencies.  It was absolutely not worth the pain.  (It took approximately 500 install/remove/configure commands to straighten out, so consider yourself forewarned!)

To get it started, I brute forced the upgrade by opening my /etc/apt/sources.list file and doing a global search and replace, changing “maverick” to “oneiric”.  At that point, you can issue the command:

sudo apt-get update; sudo apt-get dist-upgrade

That will get the upgrade started.  If you’re ready for 5 hours of agony, press Y.  Before your trial by fire can begin, you may have to wait about while the nearly 1.3Gb of updated packages download. It took about 2 hours for me.

At first, the process will look fine, installing packages quickly, with the update manager doing what it should. However, at some point, the whole upgrade crapped out on me with unresolvable dependencies.  At this point, I used two tools:

sudo apt-get dist-upgrade

and

sudo dpkg --install [deb filename]
sudo dpkg --remove [package name]

Between those two tools (and three commands), you can work through all but one of the broken package problems you’ll face. (I used synaptic to bypass that one, as mentioned below.)

Unfortunately, I don’t have any screen captures, but if you try, you’ll see a message that reads something like:

Package [name1] can't be installed, depends: [name2] (>version).
  [name2] not configured.

Each time, you’ll need to manually try to add package [name2] yourself.  The trick is that all of these files are already downloaded, so you can add them relatively easily by moving into the directory in which apt downloaded them:

cd /var/cache/apt/archives/

The (not so hard) challenge is to figure out the name of the .deb file you need to install – which is surprisingly easy.  Once you’re in the apt archives directory, all of the packages will start with the name of the package as described by the error message above.

The bigger problem is that the dependency relations can be pretty deep.  Sometimes trying to install one package will result in another message that it, in turn, depends on another package.  One by one, you have to follow each lead to figure out what the right package is, and worse, the right order to install them in.

The bigger issue is that periodically, you will find that there is a set of dependencies that you just can’t resolve, in which case, you need to remove the blocking package from the system, install the new package, then re-install the new version of the one you’ve just un-installed.  For instance, I found I had to remove the kubuntu-desktop package to get some of the kdebase packages to install.  It’s not a big deal to do this (although you need to be aware you can accidentally remove a lot of important stuff if you aren’t careful), but you’ll need to be aware of what you remove so you can add it back if (and only if!) there is an updated package for it in the archives directory.

I also found, at one point, that there were packages I could not resolve (icedtea-6-jre-jamvm and openjdk-6-jre, if I remember right) and ended up turning to synaptic. There was a simple “resolve broken packages” command or some such. That was able to get around the set I couldn’t fix manually. I still have no idea how. However, when Synaptic works, it can finish an install for you – while it can’t/doesn’t fix everything, it can be an incredibly useful tool for navigating some complex dependency nightmares.

Periodically, you’ll also discover that you have a bunch of things that are installed, but not configured because their dependencies weren’t satisfied.  You can either re-install them manually, or use the command:

sudo dpkg --configure -a

that will reconfigure any packages in that state.

Other notes – I had to remove xserver-xorg-video-all and xserver-xorg-video-nouveau to get X to work again, as well as blacklist the nouveau driver.  You can see my post here, where I describe it in more detail.  I also had to reinstall kubuntu-desktop at the end, as it was removed earlier.

Finally, I also found I needed to recompile my kernel modules (eg, for nvida-current)at some point when I did a further dist-upgrade.  For those who are familiar with the process, I discovered this on a later reboot when x refused to restart and had to drop to the terminal to make that change.

To give the complete picture, even now, not everything is now working.  Adjusting KDE plasma panels can cause significant instability in compiz (driving it up to 100% cpu), and I had to reset the compiz settings to default before compiz would stop becoming unstable. (I have a diff between the before and after settings, and nothing jumps out at me other than the active plugins and the s0_outputs resolution:

< s0_active_plugins = core;bailer;detection;composite;opengl;imgjpeg;
            debugspew;compiztoolbox;decor;vpswitch;mousepoll;resize;
            commands;imgpng;imgsvg;staticswitcher;move;dbus;resizeinfo;
            regex;text;wobbly;place;session;animation;kdecompat;fade;
            cube;rotate;expo;scale;scaleaddon;
---
> s0_active_plugins = core;bailer;composite;opengl;debugspew;decor;
            cube;kdecompat;vpswitch;resize;imgpng;rotate;move;scale;
            snap;place;

and

< s0_outputs = 1680x1050+0+0; 
--- 
> s0_outputs = 640x480+0+0;

I can’t see that helping anyone, but perhaps one of the packages previously enabled was the cause of the instability.

Hopefully the above is useful to someone else, either by pointing you in the right directions if you follow my footsteps, OR, by keeping someone from following. Either one would be a good outcome!

Good luck, and happy migrating.

Compiz running on Ubuntu 11.10 (Oneiric)

NOTE: October 18, 2011.  I see that this page is the top hit on my blog this week.  I think if you’re looking for more up to date information, you should see my current opinion on Compiz on Kubuntu here.

After months of waiting for things to be resolved with the nvidia drivers and such, I finally buckled down to see if I could get compiz running again smoothly on Ubuntu 11.10.  Happily, I can report that the answer is yes!

Using the latest packages, the trick was to follow some advice in this thread.

First, you need to blacklist the nouveau driver.  It hasn’t been working for me, and it has been preventing the nvidia driver from being loaded.

sudo nano /etc/modprobe.d/blacklist.conf

add “blacklist nouveau” to the end of the file, then save and close it.

Next, you do need to update initramfs and  reinstall the nvidia-current driver:

sudo update-initramfs -u -v
sudo apt-get install --reinstall nvidia-current

From that point on, a simple reboot (and making sure that your xorg.conf actually uses the nvidia driver) should be enough to get you back into Xorg with the Nvidia binary blob… and by opening a terminal and typing:

compiz --replace

You should be good to go again. Things appear to be stable, but I’m sure this will need a lot more testing before I switch back to compiz/kde/nvidia as my default setup.

Random OpenOffice 3.2 tip

Just in case I ever come across this problem again.

If OpenOffice 3.2 repeatedly crashes on dialog boxes for open/save/insert image/etc, the fix is to install a single package that seems to import the proper dialog box tool kit: openoffice.org-gtk

This is probably something that only happens to Kubuntu users, as openoffice.org-gtk doesn’t sound like a package you’d need on kde.

Anyhow, if you’re using kubuntu, the fix is as simple as:

sudo apt-get install openoffice.org-gtk

And instantaneously, everything works again.

A few notes on Ubuntu 11.04

Right off the bat, I have to say that I’ve been using 11.04 on one of my computers since one of the earliest alpha releases, so I’ve had the opportunity to watch it mature.  It’s something I often do – pick one computer, and use it to test out the new versions of ubuntu, upgrading the packages daily to follow along with the progress of the development. It’s usually a rewarding process, and I enjoy fixing bugs and learning how the operating system components fit together.

This was one of the few times that it was a disappointment for me.

Normally, I’ll start the process a month after the newest release, spending about two months using highly unstable versions, which then improve over time so that nearly all of the bugs are gone about a month before the official release.  This time, the release has come and gone, and my computer still doesn’t feel particularly useful.

Unlike the majority of Ubuntu users, I’m not using the default Ubuntu with Gnome window manager (or is it unity, now?), but rather KDE + Compiz.  Unfortunately, while the KDE version of Ubuntu (Kubuntu) doesn’t appear to get nearly as much attentention from the devs, it usually does march in lockstep, giving reasonable releases that come together at the last minute.  Unusually, compiz just never did come back together for me.  Perhaps it has to do with the upgrades to the xorg packages, but something is just not right when using compiz.

The symptoms are frequent restarts and crashes of the windowing system, in which the CPU usage of compiz soars to 100% and fails to respond to anything short of a the “killl -9” command.  Unfortunately, that makes compiz totally unuseable.  Ironically, it seems to happen only when closing windows, which seems like a strange place for a bug.

The workaround, if it’s actually a work around, is to completely disable compiz, which means switching back to the default KDE window manager.  Fortunately, it’s stable, so at least I can use the computer.  Unfortunately, I find Compiz to be a significant boost to my productivity, with the rotating cube desktop, the scale plugin, etc.  All of those things really make the experience on the computer, so turning all of them off it just unapealing.

In any case, for now, I’ll be staying with 10.10 on my laptops and production computers until compiz is fixed, with the possibility of jumpping straight to 11.10 as soon as that begins if I see compiz gets a bit more of the attention it deserves.

Your milage may vary, but 11.04 isn’t likely to make an appearance on my main computers anytime soon.