Wednesday, July 17, 2013

Parsing an Apache log and splitting the output into different files

Working on a project at work debugging why some requests end up with a 503 HTTP status code, I needed to split up apache logs so that I could parse them with GnuPlot.

This snippet will read multiple apache logfiles, filter out everything except status codes and the timestamp, and then split the output into different files depending on what status code it is.

Monday, May 27, 2013

Extract all files in a Quake PAK archive

This is for Linux, and my variety is Ubuntu

sudo apt-get install unmass
unmass -e pak0.pak '*'

That did not work. I wrote my own pak extractor. Follow the instructions here: https://github.com/mattissf/UnPAK

Install Cyanogenmod 10.1 on HTC One X

My setup
Be warned, this will void your warranty, and could render your phone useless / bricked. Continue at your own peril.

Before you start this procedure, you have to unlock the bootloader. I have described this process in an earlier post.

After you have downloaded the Android SDK, open a shell and navigate to the "platform-tools" directory. It contains an executable called "fastboot" that we will use later on.

First download Clockwork ROM manager from http://clockworkmod.com/rommanager. Put the file (in this case recovery-clockwork-5.8.4.0-endeavoru.img) in the "/tmp" directory.

Then download Cyanogenmod 10.1 and place it in "/tmp". At the time of this writing, the newest build for my HTC One X is a release candidate that I found here: http://wiki.cyanogenmod.org/w/Endeavoru_Info

Open up the Cyanogenmod zip file and extract the file called boot.img to "/tmp".

Now you want to transfer the zip file you downloaded to the internal SD card. This is easier said than done, since the device is not recognized as a mass USB storage device.

This info is specific for Ubuntu 12.04 and may vary on other installations. Start with opening two shells.

Shell #1:

sudo apt-get install golang-go libusb-1.0-0-dev
mkdir /tmp/mtpfs
cd /tmp/mtpfs
export GOPATH=/tmp/mtpfs
go get github.com/hanwen/go-mtpfs

mkdir /tmp/mtpfs/onex
/tmp/mtpfs/bin/go-mtpfs /tmp/mtpfs/onex

Shell #2:

mkdir "/tmp/mtpfs/onex/Internal storage/roms"
cp "/tmp/cm-10.1.0-RC2-endeavoru.zip" "/tmp/mtpfs/onex/Internal storage/roms/"
fusermount -u /tmp/mtpfs/onex

You have now copied Cyanogenmod to the phones internal storage, and you are ready to continue.

First we have to boot the phone into the bootloader. This is done by holding down the power button and the volume up button when the phone is starting.

When you are presented with an old school menu, scroll to "Fastboot" using the volume up/down keys, and select it with the power button.

Alternatively, you can just run "./adb reboot-bootloader" which I find much easier.

First we flash in the new bootloader with "sudo ./fastboot flash boot /tmp/boot.img". Not sure if it is required, but after I did this, I selected the option to reboot the bootloader.

When the bootloader is up again, go into fastboot mode and flash in the recovery image we downloaded earlier:

"sudo ./fastboot flash recovery /tmp/Downloads/recovery-clockwork-5.8.4.0-endeavoru.img"

When that is done, we boot the phone into recovery mode by selecting "HBOOT" in the fastboot options.  You are now presented with multiple choices. Select "RECOVERY" and press the power button.

When you are presented with the ClockworkMod Recovery menu, it is wise to select backup.

Then "Wipe data/factory reset", "Wipe cache partition" and "Advanced/Wipe Dalvik Cache".

We are now ready to install Cyanogenmod. Select "install zip from sdcard" and "choose zip from sdcard." Navigate to the "roms" directory and select the cyanogenmod zip file we transferred earlier.

Installation will now continue. When you are done, go back and select "reboot system now." The phone should now boot up into Cyanogenmod. You are done!

Unlocking the bootloader of my HTC One X

My setup
Be warned, this will void your warranty, and could render your phone useless / bricked. Continue at your own peril.

After you have downloaded the Android SDK, open a shell and navigate to the "platform-tools" directory. It contains an executable called "fastboot" that we will use later on.

First we have to boot the phone into the bootloader. This is done by holding down the power button and the volume up button when the phone is starting.

When you are presented with an old school menu, scroll to "Fastboot" using the volume up/down keys, and select it with the power button.

Your phone is now in fastboot mode.

From your shell, type in "sudo ./fastboot oem get_identifier_token." You should get a long string of text. This is your unique token. Save it for later.

Now go to "http://www.htcdev.com/" and select "Unlock Bootloader." In the dropdown, select "All Other Supported Devices." You can skip all of this until you reach the input box where you should paste in the token you saved in the last step. If you get a failure, read the instructions on the HTC webpage that explains very detailed what the token should look like.

You should receive an email from HTC with an attachment, "Unlock_code.bin" in my case. Download the attachment and put it in the /tmp directory.

In your shell, type in "sudo ./fastboot flash unlocktoken /tmp/Unlock_code.bin" which should prompt you on the phone if you want to unlock the bootloader.

Select "Yes" and press the power button. Your phone will now reboot, and your bootloader is now unlocked.

Tuesday, May 21, 2013

How to enable debugging on Garmin ANT Agent

I had a problem where the Garmin ANT Agent would connect to my Forerunner 910XT and try and sync up with new activities, only to discover that it would fail with "Downloading fitness data failed" error message.

I scourged the internet high and low to find a solution, but most of them involved resetting my watch to factory defaults. Which I did not want.

After a while, I found a recipe for enabling debugging on a Mac here. But this did not help me, as I am using a Windows 7 machine.

So fiddling a bit about, I managed to get debugging up and running by doing these steps:
  1. Click "Start Menu" and type in "regedit.exe"
  2. Go to "HKEY_CURRENT_USER\Software\Garmin\Garmin ANT Agent(tm)\Settings"
  3. Create a new entry of type "Binary Value" with the name "ShowDebug"
  4. Set the value to "01"
It should look something like this:


Now restart your ANT Agent and there should be loads of debug files located in the folder C:\Users\mattis\AppData\Local\Temp\ANTAgent\

"Unfortunately", right after I enabled debugging, the problem fixed itself, so I never had to actually analyze the logs.

Monday, December 6, 2010

Apache Segmentation Fault because of OCI / Zend in a non-standard environment

We are running Apache in a jailed environment in the context of a user that has no home directory, so the OCI8 client that comes with Zend segmentation faults when it tries to write diagnostic files.

By using GDB I made an educated guess that OCI8 is trying to figure out the home directory and probably crashes because there is no corresponding home directory in my /etc/passwd file.

GDB output:

Core was generated by `/usr/sbin/httpd -k start'.
Program terminated with signal 11, Segmentation fault.
#0  0x00002b6a1a5e51be in skgpgetinfo () from /usr/lib64/libclntsh.so.11.1

#0  0x00002b6a1a5e51be in skgpgetinfo () from /usr/lib64/libclntsh.so.11.1
#1  0x00002b6a1a74d89b in dbgroud_oradiag_user_dir () from /usr/lib64/libclntsh.so.11.1
#2  0x00002b6a1a74d067 in dbgrgad_get_adrbase_directory () from /usr/lib64/libclntsh.so.11.1
#3  0x00002b6a1a76b81d in kpeDbgGetInitFileParms () from /usr/lib64/libclntsh.so.11.1
#4  0x00002b6a1a76b5fe in kpeDbgGetInitFileParmsAndInitDBGC () from /usr/lib64/libclntsh.so.11.1
#5  0x00002b6a1a187c6d in nlstddd_do_alter_diag () from /usr/lib64/libclntsh.so.11.1
#6  0x00002b6a1a18594d in nlstddt_do_alter_trace () from /usr/lib64/libclntsh.so.11.1
#7  0x00002b6a1bbf699e in nlstdggo () from /usr/lib64/libclntsh.so.11.1
#8  0x00002b6a1bbf66af in nlstdgg () from /usr/lib64/libclntsh.so.11.1
#9  0x00002b6a1a23b5fb in nigini2 () from /usr/lib64/libclntsh.so.11.1
#10 0x00002b6a1a76be3c in kpeDbgGetNPDGlobal () from /usr/lib64/libclntsh.so.11.1
#11 0x00002b6a1a76bd10 in kpeDbgTLSInit () from /usr/lib64/libclntsh.so.11.1
#12 0x00002b6a1bd1c331 in kpummTLSGET1 () from /usr/lib64/libclntsh.so.11.1
#13 0x00002b6a1a76bfd3 in kpeDbgProcessInit () from /usr/lib64/libclntsh.so.11.1
#14 0x00002b6a1a76a457 in kpummpin () from /usr/lib64/libclntsh.so.11.1
#15 0x00002b6a1a0dcd13 in kpuenvcr () from /usr/lib64/libclntsh.so.11.1
#16 0x00002b6a1acfbdc1 in OCIEnvNlsCreate () from /usr/lib64/libclntsh.so.11.1
#17 0x00002b6a19a12a3b in php_oci_do_connect_ex () from /usr/local/zend/lib/php_extensions/oci8.so
#18 0x00002b6a19a12fcf in php_oci_do_connect () from /usr/local/zend/lib/php_extensions/oci8.so
#19 0x00002b6a19a23953 in zif_oci_connect () from /usr/local/zend/lib/php_extensions/oci8.so
#20 0x00002b6a131cdd69 in ?? () from /usr/local/zend/lib/apache2/libphp5.so
#21 0x00002b6a131a458e in execute () from /usr/local/zend/lib/apache2/libphp5.so
#22 0x00002b6a13180e99 in zend_execute_scripts () from /usr/local/zend/lib/apache2/libphp5.so
#23 0x00002b6a1312f7f8 in php_execute_script () from /usr/local/zend/lib/apache2/libphp5.so
#24 0x00002b6a1320887d in ?? () from /usr/local/zend/lib/apache2/libphp5.so
#25 0x00002b6a084cda4a in ap_run_handler ()
#26 0x00002b6a084d0ed8 in ap_invoke_handler ()
#27 0x00002b6a084db938 in ap_process_request ()
#28 0x00002b6a084d8b70 in ?? ()
#29 0x00002b6a084d4cd2 in ap_run_process_connection ()
#30 0x00002b6a084df789 in ?? ()
#31 0x00002b6a084dfa1a in ?? ()
#32 0x00002b6a084dfad0 in ?? ()
#33 0x00002b6a084e07bb in ap_mpm_run ()
#34 0x00002b6a084bae48 in main ()

This was fixed by disabling diagnostics in OCI by setting $ORACLE_HOME env to /tmp and then put "DIAG_ADR_ENABLED=OFF" into $ORACLE_HOME/network/admin/sqlnet.ora file.

Thursday, March 25, 2010

Installing bare bones Eclipse

If you have ever wanted Eclipse without Java, C++, PHP and so on, you can get a bare bones Eclipse here:

http://download.eclipse.org/eclipse/downloads/

To get to the download link, follow these steps:

  • Click the latest release
  • On the next page, on the left menu, click "Platform Runtime Binary"
  • Click the appropriate link for your platform