Main

September 8, 2013

Patching the Linux Kernel to install the Xin-Mo Dual Arcade driver on Ubuntu 13.04

These are instruction for Ubuntu. I also have instructions for the Raspberry Pi.

The Xin-Mo Dual Arcade is a USB interface for up to two arcade joysticks and 22 buttons. It’s a cool little device if you want to build an arcade controller for your computer or Raspberry Pi. You can get one from Ultracabs.

Unfortunately there is a bug in the Xin-Mo Dual Arcade: the values it sends for the up and left positions of the joysticks are out of range with the values it announces it would send, and Linux therefore ignores these values (it’s its right). Therefore the joystick do nothing in the up and left position.

I wrote a driver to fix that problem, as a patch to the Linux Kernel. A few people asked me for help on how to install it, hence this article.

Cliché Disclaimer: Use these instructions at your own risks. Backup your computer before you start. I am not responsible for any data loss or fried computers caused by following these instructions.

Ubuntu

I used Ubuntu 13.04, but it should be the same for the other affected versions (12.04, 12.10). Everything here happens in the terminal, make sure you’re logged in as an administrator (typically the user you created when you installed Ubuntu). Whenever a command that starts with sudo asks for your password, use the password for that administrator user. These instructions were derived from the BuildYourOwnKernel page on the Ubuntu Wiki.

Observing the problem

First we’re going to install evtest, a tool to test the joystick:

sudo apt-get update
sudo apt-get install evtest

To try it out, connect your joystick and run the program:

evtest

It will list devices and ask you to select your joystick. Find the line that says Xin-Mo Xin-Mo Dual Arcade, type the number next to /dev/input/event (4 in this example) and Enter.

No device specified, trying to scan all of /dev/input/event*
Not running as root, no devices may be available.
Available devices:
/dev/input/event4:  Xin-Mo Xin-Mo Dual Arcade
Select the device event number [0-5]: 4

Now when you move the joysticks and press buttons, it will print lines like these (for a button):

Event: time 1378629038.490828, type 4 (EV_MSC), code 4 (MSC_SCAN), value 90001
Event: time 1378629038.490828, type 1 (EV_KEY), code 288 (BTN_TRIGGER), value 1
Event: time 1378629038.490828, -------------- SYN_REPORT ------------
Event: time 1378629038.618674, type 4 (EV_MSC), code 4 (MSC_SCAN), value 90001
Event: time 1378629038.618674, type 1 (EV_KEY), code 288 (BTN_TRIGGER), value 0
Event: time 1378629038.618674, -------------- SYN_REPORT ------------

or these (for a joystick):

Event: time 1378629075.546079, type 3 (EV_ABS), code 1 (ABS_Y), value 1
Event: time 1378629075.546079, -------------- SYN_REPORT ------------
Event: time 1378629075.740176, type 3 (EV_ABS), code 1 (ABS_Y), value 0
Event: time 1378629075.740176, -------------- SYN_REPORT ------------

and nothing when you move the joysticks up or left.

Fixing the problem

Downloading the sources

First we need to install the programs needed to build the kernel1:

sudo apt-get build-dep linux-image-$(uname -r)

Then we can download the source code for the kernel:

apt-get source linux-image-$(uname -r)

It will download and extract the following files to the current directory (the numbers can vary):

linux-3.8.0
linux_3.8.0-30.44.diff.gz
linux_3.8.0-30.44.dsc
linux_3.8.0.orig.tar.gz

Applying the patch

linux-3.8.0 is the directory where the source code was extracted. Enter it:

cd linux-3.8.0

Before we start, we need to install a tool called fakeroot2:

sudo apt-get install fakeroot

Then download the patch using wget:

wget 'http://ithink.ch/blog/files/xin-mo/0001-hid-Add-new-driver-for-non-compliant-Xin-Mo-devices.patch'

and apply it:

patch -p1 < 0001-hid-Add-new-driver-for-non-compliant-Xin-Mo-devices.patch

It should display something like this:

patching file drivers/hid/Kconfig
Hunk #1 succeeded at 697 with fuzz 2 (offset -46 lines).
patching file drivers/hid/Makefile
Hunk #1 succeeded at 108 (offset -2 lines).
patching file drivers/hid/hid-core.c
Hunk #1 succeeded at 1738 (offset 2 lines).
patching file drivers/hid/hid-ids.h
Hunk #1 succeeded at 849 (offset -38 lines).
patching file drivers/hid/hid-xinmo.c

Building the kernel

Run the following commands to start the build:

fakeroot debian/rules clean
fakeroot debian/rules binary-headers binary-generic

After a short while, it should pause to ask you about enabling a new driver:

Xin-Mo non-fully compliant devices (HID_XINMO) [N/m/?] (NEW)

That’s the one we’re going through all this trouble for, so tell it to install the module by typing m and Enter.

Then the actual compilation will begin. It will take a while.

Installing the new kernel

Exit the linux-3.8.0 directory:

cd ..

Four debian packages were created (again, the numbers will vary):

linux-headers-3.8.0-30_3.8.0-30.44_all.deb
linux-headers-3.8.0-30-generic_3.8.0-30.44_amd64.deb
linux-image-3.8.0-30-generic_3.8.0-30.44_amd64.deb
linux-image-extra-3.8.0-30-generic_3.8.0-30.44_amd64.deb

Install them one by one:

sudo dpkg -i linux-headers-3.8.0-30_3.8.0-30.44_all.deb
sudo dpkg -i linux-headers-3.8.0-30-generic_3.8.0-30.44_amd64.deb
sudo dpkg -i linux-image-3.8.0-30-generic_3.8.0-30.44_amd64.deb
sudo dpkg -i linux-image-extra-3.8.0-30-generic_3.8.0-30.44_amd64.deb

And then reboot:

sudo reboot

Test the patch

Run evtest again, like above, and enjoy the result:

Event: time 1378634592.175998, type 3 (EV_ABS), code 1 (ABS_Y), value -1
Event: time 1378634592.175998, -------------- SYN_REPORT ------------
Event: time 1378634592.335920, type 3 (EV_ABS), code 1 (ABS_Y), value 0
Event: time 1378634592.335920, -------------- SYN_REPORT ------------
Event: time 1378634593.613888, type 3 (EV_ABS), code 0 (ABS_X), value -1
Event: time 1378634593.613888, -------------- SYN_REPORT ------------
Event: time 1378634593.809001, type 3 (EV_ABS), code 0 (ABS_X), value 0
Event: time 1378634593.809001, -------------- SYN_REPORT ------------
Event: time 1378634595.165686, type 3 (EV_ABS), code 3 (ABS_RX), value -1
Event: time 1378634595.165686, -------------- SYN_REPORT ------------
Event: time 1378634595.325688, type 3 (EV_ABS), code 3 (ABS_RX), value 0
Event: time 1378634595.325688, -------------- SYN_REPORT ------------
Event: time 1378634595.933826, type 3 (EV_ABS), code 2 (ABS_Z), value -1
Event: time 1378634595.933826, -------------- SYN_REPORT ------------
Event: time 1378634596.093610, type 3 (EV_ABS), code 2 (ABS_Z), value 0
Event: time 1378634596.093610, -------------- SYN_REPORT ------------

If these instructions helped you, I would appreciate it if you could send me a couple of pictures of your set-up so I can compile them on a web page. Drop me a note on Twitter.

1. And we already need to deviate from the instructions on the Ubuntu Wiki, because they tell you to download the sources first, but it will fail on a fresh install because we’re missing some tools. That shows us how carefully those instructions are written. 

2. Second deviation, they forgot about installing fakeroot

August 19, 2012

Time Machine Waiting for Index to Be Ready

My Time Machine backup (on OS X 10.7) was stuck on Waiting for index to be ready (100) for a long time (it kept logging this once every minute for 20 minutes).

I ended up killing the mds process. I was treated with an Indexer unavailable (200), then another Waiting for index to be ready, then the backup completed successfully:

19.08.2012 14:25:34.270 com.apple.backupd: Waiting for index to be ready (100)
19.08.2012 14:26:34.352 com.apple.backupd: Waiting for index to be ready (100)
19.08.2012 14:27:34.429 com.apple.backupd: Waiting for index to be ready (100)
19.08.2012 14:28:13.502 com.apple.backupd: Indexer unavailable (200)
19.08.2012 14:28:35.520 com.apple.backupd: Waiting for index to be ready (100)
19.08.2012 14:30:14.957 com.apple.backupd: Copied 2944 files (5.2 MB) from volume filemate.
19.08.2012 14:30:15.574 com.apple.backupd: Copied 2955 files (5.2 MB) from volume 500GB.
19.08.2012 14:30:32.853 com.apple.backupd: Backup completed successfully.

October 2, 2011

Installer Redmine sur Mac OS X 10.6: La façon correcte

Oui, c’est un titre plutôt prétentieux étant donné que c’est la première fois que je réussis à installer une application Rails, et non, cela ne me dérange pas. C’était agréable comme un coup de pied au cul alors je dois documenter la façon dont j’y suis parvenu et cela pourrait rendre service à d’autres, alors si ça vous dérange, lisez-le comme ‘la façon correcte pour moi’.

Redmine (1.2.1 du 2011-07-11 au moment d’écrire ces lignes) est sacrément regardant en ce qui concerne ses prérequis: les gems (et aussi Ruby et RubyGems) ne doivent pas être trop anciens, certains ne peuvent pas être trop récents, et l’un d’eux (i18n) ne peut même pas être d’une version différente que celle spécifiée (0.4.2).

La page d’installation de Redmine indique qu’il nécessite:

  • Ruby 1.8.6 ou 1.8.7, pas 1.9;
  • RubyGems 1.3.7 ou plus récent, mais pas la 1.5.0 ni la 1.7.0 ou supérieure, parce que Rails 2.3.11 ne fonctionne pas avec celles-ci;
  • Rails 2.3.11 (puis plus loin mentionne que la 2.3.14 fonctionne aussi);
  • Rack 1.1.1;
  • Rake 0.8.7 (pas la 0.9.x parce qu’elle n’est pas supportée par Rails);
  • i18n 0.4.2 (la version 0.6.0 actuelle causera une erreur parce que la 0.4.2 est introuvable).

De plus, Redmine est distribué avec Rails 2.3.11 dans le dossier ‘vendor’, et utilisera celui-là à moins que vous ne l’en supprimiez et changiez la version désirée dans config/environment.rb. Ceci est élégamment documenté sur la page d’installation par une phrase très explicative: “read config/environment.rb first”. Simplement changer la version de Rails dans ce fichier n’aidera pas: si Redmine trouve Rails dans son dossier ‘vendor’, il utilisera celui-là.

La réalité des prérequis de Redmine est légèrement différente. Rake 0.9.x est supporté par Rails 2.3.14 (il installe la 0.9.2 avec ses dépendances), mais RubyGems 1.7.0 ou plus récent ne l’est quand-même pas. Alors pour réduire la quantité de software obsolète et buggué que l’installe et les soucis qui s’ensuivent, je vais utiliser Rails 2.3.14.

Utiliser RVM

Ruby Version Manager, ou RVM, permet d’utiliser une version de Ruby et une gamme de gems différents pour des projets différents, sans installer lesdits gems au niveau du système. I est similaire au virtualenv de Python. À mon avis, il est absolument vital de l’employer dès le départ. Vous ne savez pas quels autres programmes vous pourriez vouloir installer, programmes qui pourraient avoir des prérequis aussi délicats que Redmine mais dans des version différentes. 

RVM se trouve à l’adresse http://beginrescueend.com/ et s’installe comme ceci (apparemment il nécessite git, ce qui est un peu gonflant si vous ne vous en servez pas. Mais vous devriez l’essayer, couplé à la version d’évaluation de Tower, c’est fantastique une fois que vous commencez à piger):

$ bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)

Ajoutez ensuite ceci à votre fichier.bash_profile ou .zshrc (vous voulez clairement passer à zsh et utiliser oh-my-zsh, parce qu’ils sont fantastiques également), comme indiqué par le script d’installation (personnalisé pour votre  $HOME):

# This loads RVM into a shell session.
[[ -s "/Users/olivier/.rvm/scripts/rvm" ]] && source "/Users/olivier/.rvm/scripts/rvm"

Ensuite nous devons installer la dernière version de Ruby pré-1.9 pour utiliser avec RVM. RVM le compile depuis les sources, mais cela ne prends pas trop longtemps (et si vous trouvez que si, faites un saut sur 9gag.com). On peut lister les version disponibles de Ruby et ensuite installer celle que l’on veut, c’est à dire la 1.8.7 (qui donnera ruby-1.8.7-p352), comme suit:

$ rvm list known

# MRI Rubies
[ruby-]1.8.6[-p420]
[ruby-]1.8.6-head
[ruby-]1.8.7[-p352]
[ruby-]1.8.7-head
[ruby-]1.9.1-p378
[ruby-]1.9.1[-p431]
[ruby-]1.9.1-head
[ruby-]1.9.2-p180
[ruby-]1.9.2[-p290]
[ruby-]1.9.2-head
[ruby-]1.9.3[-preview1]
[ruby-]1.9.3-head
ruby-head
...

$ rvm install 1.8.7
...
$ rvm use 1.8.7

Une fois fait, nous allon créer un gemset pour Redmine, où RubyGems installera les gems dont nous avons besoin, dans les versions dont nous avons besoin, sans interférer avec l’installation système.

$ rvm gemset create redmine
'redmine' gemset created (/Users/olivier/.rvm/gems/ruby-1.8.7-p352@redmine).
$ rvm gemset use redmine

Notes:

  1. La version de Ruby et le gemset employés sont liés à la session shell, ce qui fait que vous devez les changer à nouveau à chaque nouvelle session. Cela peut être effectué en une étape:

    $ rvm use 1.8.7@redmine
    Using /Users/olivier/.rvm/gems/ruby-1.8.7-p352 with gemset redmine
    
  2. L’environnement en cours d’utilisation peut être vérifié avec rvm-prompt:

    $ rvm-prompt        
    ruby-1.8.7-p352@redmine
    
  3. Vous pouvez repasser au Ruby système avec:

    $ rvm use system
    Now using system ruby.
    
  4. Il n’y a pas besoin de sudo pour installer des gems dans un gemset RVM, parce que tous les fichiers sont contenus dans votre dossier Home.

  5. Quand un gemset est en cours d’utilisation, les commandes Rake doivent être préfixées par rvm:

    $ rvm rake bla bla
    

Installer les gems

Tout d’abord, accélérons l’installation des gems d’un facteur 10, comme suggéré par The Budding Rubbyist. Ajoutez ceci à votre fichier ~/.gemrc, en le créant si nécessaire:

gem: --no-ri --no-rdoc

Cela ommettra la documentation (que nous ne lirons jamais, étant donné que nous ne sommes intéressés qu’à installer des programmes Ruby, pas à en développer) à l’installation des gems, une étape inexplicablement très lente, puisqu’elle dure généralement plus longtemps que l’étape “Building native extensions. This could take a while…” de certains gems.

Ensuite, assurons-nous que nous utilisons bien le Ruby et le gemset créés précédemment:

$ rvm use 1.8.7@redmine

Installons ensuite Rails 2.3.14 et i18n 0.4.2 (Rails installe i18n 0.6.0 mais souvenez-vous, Redmine ne la verrait pas même si on la bourrait dans son crâne épais avec une [we]brique):

gem install rails -v=2.3.14
gem install i18n -v=0.4.2

Note: pour déterminer que c’était Rails 2.3.14 que je voulais, j’ai listé toutes les versions disponibles et choisi la 2.3.x la plus récente:

$ gem query  --all --remote -n '^rails$'

*** REMOTE GEMS ***

rails (3.1.0, 3.0.10, 3.0.9, 3.0.8, 3.0.7, 3.0.6, 3.0.5, 3.0.4, 3.0.3, 3.0.2, 3.0.1, 3.0.0, 2.3.14, 2.3.12, 2.3.11, 2.3.10, 2.3.9, 2.3.8, 2.3.7, 2.3.6, 2.3.5, 2.3.4, 2.3.3, 2.3.2, 2.2.3, 2.2.2, 2.1.2, 2.1.1, 2.1.0, 2.0.5, 2.0.4, 2.0.2, 2.0.1, 2.0.0, 1.2.6, 1.2.5, 1.2.4, 1.2.3, 1.2.2, 1.2.1, 1.2.0, 1.1.6, 1.1.5, 1.1.4, 1.1.3, 1.1.2, 1.1.1, 1.1.0, 1.0.0, 0.14.4, 0.14.3, 0.14.2, 0.14.1, 0.13.1, 0.13.0, 0.12.1, 0.12.0, 0.11.1, 0.11.0, 0.10.1, 0.10.0, 0.9.5, 0.9.4.1, 0.9.4, 0.9.3, 0.9.2, 0.9.1, 0.9.0, 0.8.5, 0.8.0)

La commande list est plus simple mais affichera tout ce qui commence par rails, ce qui est un peu gênant:

$ gem list --all --remote rails

*** REMOTE GEMS ***

rails (3.1.0, 3.0.10, 3.0.9, 3.0.8, 3.0.7, 3.0.6, 3.0.5, 3.0.4, 3.0.3, 3.0.2, 3.0.1, 3.0.0, 2.3.14, 2.3.12, 2.3.11, 2.3.10, 2.3.9, 2.3.8, 2.3.7, 2.3.6, 2.3.5, 2.3.4, 2.3.3, 2.3.2, 2.2.3, 2.2.2, 2.1.2, 2.1.1, 2.1.0, 2.0.5, 2.0.4, 2.0.2, 2.0.1, 2.0.0, 1.2.6, 1.2.5, 1.2.4, 1.2.3, 1.2.2, 1.2.1, 1.2.0, 1.1.6, 1.1.5, 1.1.4, 1.1.3, 1.1.2, 1.1.1, 1.1.0, 1.0.0, 0.14.4, 0.14.3, 0.14.2, 0.14.1, 0.13.1, 0.13.0, 0.12.1, 0.12.0, 0.11.1, 0.11.0, 0.10.1, 0.10.0, 0.9.5, 0.9.4.1, 0.9.4, 0.9.3, 0.9.2, 0.9.1, 0.9.0, 0.8.5, 0.8.0)
rails-3-settings (0.1.1)
rails-action-args (0.1.1, 0.1.0)
rails-admin (0.0.0)
rails-and-solid (0.9.1, 0.9.0)
...

Je voulais utiliser SQLite (je n’ai pas pu compiler le gem mysql contre le MySQL de Zend Server, et je m’en fiche — oh, c’est notre shortstop), alors j’ai installé le gem sqlite3:

gem install sqlite3

Nous devons aussi downgrader RubyGems depuis la dernière version (1.8.10 au moment d’écrire ces lignes) vers une version qui est au moins la 1.3.7 mais pas la 1.5.0, ni la 1.7.0 ou plus récente. La version que nous voulons peut être déterminée de la même façon que pour Rails, en demandant le gem rubygems-update:

$ gem list --all --remote  rubygems-update

*** REMOTE GEMS ***

rubygems-update (1.8.10, 1.8.9, 1.8.8, 1.8.7, 1.8.6, 1.8.5, 1.8.4, 1.8.3, 1.8.2, 1.8.1, 1.8.0, 1.7.2, 1.7.1, 1.7.0, 1.6.2, 1.6.1, 1.6.0, 1.5.3, 1.5.2, 1.5.0, 1.4.2, 1.4.1, 1.4.0, 1.3.7, 1.3.6, 1.3.5, 1.3.4, 1.3.3, 1.3.2, 1.3.1, 1.3.0, 1.2.0, 1.1.1, 1.1.0, 1.0.1, 1.0.0, 0.9.5, 0.9.4, 0.9.3, 0.9.2, 0.9.1, 0.9.0, 0.8.11, 0.8.10, 0.8.8, 0.8.6, 0.8.5, 0.8.4, 0.8.3)

Nous voulons donc la 1.6.2:

$ gem update --system 1.6.2
...
$ gem -v
1.6.2

Noter que d’anciennes versions de la commande gem, comme la 1.3.7 recommendée par Redmine, ne supportent pas de spécifier la version dans la commande gem update --system. Dans ce cas, vous devez d’abord obtenir la dernière version et ensuite la downgrader:

$ gem update --system
$ gem update --system 1.6.2

Passons maintenant à l’installation de Redmine.

Installer Redmine

Téléchargez la dernière version stable (1.2.1 du 2011-07-11 au moment d’écrire ces lignes) et décompressez-la.

Nous devons désactiver le Rails 2.3.11 livré et dire à Redmine d’utiliser notre version 2.3.14. Dans config/environment.rb, changer ces lignes:

# Specifies gem version of Rails to use when vendor/rails is not present
RAILS_GEM_VERSION = '2.3.11' unless defined? RAILS_GEM_VERSION

en:

# Specifies gem version of Rails to use when vendor/rails is not present
RAILS_GEM_VERSION = '2.3.14' unless defined? RAILS_GEM_VERSION

Ensuite, déplacez Rails hors du dossier ‘vendor’, sinon Redmine utilisera l’ancienne version, quoique la ligne ci-dessus dise. Je le fais comme ceci:

mv vendor/rails vendor.rails.disabled

Au fait, c’est le passage “read config/environment.rb first” dans la liste des prérequis et le “when vendor/rails is not present” dans le commentaire ci-dessus qui nous indique que nous devons enlever Rails du dossier ‘vendor’. Comme je l’ai dit, une phrase très explicative.

Procédons ensuite à l’installation. Avec SQLite, database.yml a l’allure suivante:

production:
  adapter: sqlite3
  database: db/redmine.db
 
development:
  adapter: sqlite3
  database: db/redmine.db

# Warning: The database defined as "test" will be erased and
# re-generated from your development database when you run "rake".
# Do not set this db to the same as development or production.
test:
  adapter: sqlite3
  database: db/redmine_test.db

Note: Chez moi, les commandes suivantes étaient infestées d’alertes d’obsolescence comme celle-ci:

rake/rdoctask is deprecated.  Use rdoc/task instead (in RDoc 2.4.2+)
Please install RDoc 2.4.2+ to generate documentation.
WARNING: 'task :t, arg, :needs => [deps]' is deprecated.  Please use 'task :t, [args] => [deps]' instead.
    at redmine-1.2.1/lib/tasks/email.rake:170

À ma connaissance elles ne dérangent pas, à part en rendant difficile de savoir si la tâche s’est achevée avec succès ou pas.

Créez un ‘session store secret’ (n’oubliez pas de taper rvm rake au lieu de rake):

rvm rake generate_session_store

Créez la structure de la base de données:

RAILS_ENV=production rvm rake db:migrate

Insérez la configuration par défaut dans la base de données:

RAILS_ENV=production rvm rake redmine:load_default_data

Testez ensuite votre installation. Cela prend un petit moment pour démarrer. Les deux premières lignes apparaissent rapidement et vous permettent de vérifier que Redmine utilise la bonne version de Rails:

=> Booting WEBrick
=> Rails 2.3.14 application starting on http://0.0.0.0:3000

Le serveur est prêt quand vous voyez ceci:

=> Call with -d to detach
=> Ctrl-C to shutdown server
[2011-09-25 14:28:09] INFO  WEBrick 1.3.1
[2011-09-25 14:28:09] INFO  ruby 1.8.7 (2011-06-30) [i686-darwin10.8.0]
[2011-09-25 14:28:09] INFO  WEBrick::HTTPServer#start: pid=39670 port=3000

Faire tourner Redmine sous Unicorn

Avertissement: Je n’ai aucune idée de si utiliser Unicorn pour faire tourner Redmine est une bonne idée. Je sais juste que WEBrick n’est pas destiné à la production et que Passenger est un module Apache, et je préfère l’éviter si possible. Après-tout, nous sommes arrivés jusqu’ici sans nécessiter de droits administrateur.

J’ai trouvé les instructions sur Install Redmine on Ubuntu Server dans la liste des HowTo de Redmine, sauf la partie concernant le wrapper RVM, qui vient de Brandon Tilley.

Installez Unicorn, toujours sous RVM:

$ gem install unicorn

Créez un wrapper pour unicorn_rails. Le wrapper fera tourner unicorn_rails avec la bonne version de Ruby et le bon gemset:

$ rvm wrapper ruby-1.8.7-p352@redmine redmine unicorn_rails

Cela crée une commande redmine_unicorn_rails qui utilise l’environnement ruby-1.8.7-p352@redmine.

Créez le fichier de configuration d’Unicorn, par example unicorn.conf dans le dossier config de Redmine:

# /var/www/redmine/config/unicorn.conf

# Number of CPU Cores
worker_processes 2

# Same Redmine User
user "redmine", "redmine"
working_directory "/var/www/redmine"

# Listen on Both UNIX and TCP Sockets for Testing
listen "/var/www/redmine/tmp/sockets/.sock", :backlog => 64
listen 8080, :tcp_nopush => true

# Default Timeout
timeout 30

# Pid/Logging Config
pid "/var/www/redmine/tmp/pids/unicorn.pid"
stderr_path "/var/www/redmine/log/unicorn.stderr.log"
stdout_path "/var/www/redmine/log/unicorn.stdout.log"

# Preload Rails App for Performance
preload_app true

before_fork do |server, worker|
  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.connection.disconnect!
end

after_fork do |server, worker|
  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.establish_connection
end

Cela suppose que vous avez décompressé Redmine dans /var/www/redmine et que vous avez un utilisateur redmine appartenant au groupe redmine. Si nécessaire, modifiez toutes les occurrences de /var/www/redmine vers le chemin où vous avez installé Redmine et la ligne user "redmine", "redmine" avec l’utilisateur et le groupe sous lesquels vous voulez que Redmine s’exécute. Pour mes tests j’ai utilisé mon compte utilisateur, mais ce n’est certainement pas recommandé:

user "olivier", "staff"

Démarrez ensuite Unicorn en utilisant le wrapper, ce qui signifie qu’il n’est pas nécessaire de changer l’environnement RVM. Cela sera pratique si vous utilisez launchd ou un autre gestionnaire de services pour lancer Redmine:

$ sudo -u redmine redmine_unicorn_rails -c /var/www/redmine/config/unicorn.conf -D -E production

Si comme moi vous avez utilisé votre propre compte utilisateur comme test, alors le sudo n’est pas nécessaire:

$ redmine_unicorn_rails -c /var/www/redmine/config/unicorn.conf -D -E production

D’après le fichier de configuration, Unicorn tourne sur le port 8080. Si vous avez un serveur Apache sur le port 80 et désirez que Redmine apparaisse sous un préfixe donné, alors vous pouvez probablement utiliser une réécriture de type proxy pour atteindre ce but, à condition d’avoir activé mod_proxy.

Pour arrêter Unicorn, tuez le process unicorn_rails master.

Conclusion

C’est tout pour aujourd’hui, j’espère que cela a été utile. Comme je l’ai déjà mentionné, je débute avec les applications Rails, alors j’ai probablement fait des erreurs. N’hésitez pas à les signaler dans les commentaires, j’essaierai de mettre à jour ce post en conséquence. Il se peut aussi que j’ajoute quelque chose concernant l’utilisation de launchd pour exécuter Redmine sous Unicorn et de mod_proxy pour l’intégrer avec un site web existant.

September 25, 2011

Installing Redmine on Mac OS X 10.6: The Correct Way

Yes, it’s a rather pretentious title, considering that it’s my first time succeeding at installing a Rails application, and no, I don’t care. It was a pain in the ass so I need to document how I did it, and it might help other people, so if it really bothers you, read it as ‘the correct way for me’.

Redmine (1.2.1 of 2011-07-11 as of this writing) is pretty touchy about its requirements: gems (and Ruby and RubyGems as well) cannot be too old, some cannot be too recent, and one (i18n) even cannot be anything other than the specified version (0.4.2).

The Redmine installation page says it requires:

  • Ruby 1.8.6 or 1.8.7, not 1.9;
  • RubyGems 1.3.7 or higher, but not 1.5.0 and not 1.7.0 or higher, because Rails 2.3.11 fails with it;
  • Rails 2.3.11 (then later tells you that 2.3.14 works as well);
  • Rack 1.1.1;
  • Rake 0.8.7 (not 0.9.x because it’s not supported by Rails);
  • i18n 0.4.2 (the current version 0.6.0 will give you an error that 0.4.2 cannot be found).

Moreover, Redmine comes with Rails 2.3.11 bundled in the vendor directory, and will use it unless you remove it from there and change the version you want in config/environment.rb. This is neatly documented on the install page with a very thorough explanation: “read config/environment.rb first”. Just changing the version won’t help: if Redmine finds rails in its vendor directory, it will use that one.

The reality of Redmine requirements is a bit different. Rake 0.9.x is supported by Rails 2.3.14 (it installs 0.9.2 as part of its dependencies), but RubyGems 1.7.0 or higher still isn’t. So to minimise the number of obsolete and buggy software that I run and the fuss that ensues, I am going to use Rails 2.3.14.

Using RVM

Ruby Version Manager, or RVM, allows you to use a different version of Ruby and a different set of gems for different projects, without installing said gems system-wide. It is similar to Python’s virtualenv. In my opinion, it is absolutely vital to use it right from the start. You never know what other software you might want to install later, software that might have requirements as tricky as Redmine’s, yet different in the actual versions.

RVM can be found at http://beginrescueend.com/ and is installed thusly (it apparently requires git, which is kind of a bummer if you don’t use it. But you should give it a try, it’s awesome once you start grokking it):

$ bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)

Then add the following to your .bash_profile or .zshrc (obviously you want to switch to zsh and use oh-my-zsh, because they’re awesome too) as the install script will tell you (personalised with your $HOME path):

# This loads RVM into a shell session.
[[ -s "/Users/olivier/.rvm/scripts/rvm" ]] && source "/Users/olivier/.rvm/scripts/rvm"

Next we need to install the latest pre-1.9 Ruby to use with RVM. RVM will build it from source, but it doesn’t take too long (if you find it does, just head over to 9gag.com. We can list the available Ruby versions, and then install the one we want, namely 1.8.7 (that will result in ruby-1.8.7-p352):

$ rvm list known

# MRI Rubies
[ruby-]1.8.6[-p420]
[ruby-]1.8.6-head
[ruby-]1.8.7[-p352]
[ruby-]1.8.7-head
[ruby-]1.9.1-p378
[ruby-]1.9.1[-p431]
[ruby-]1.9.1-head
[ruby-]1.9.2-p180
[ruby-]1.9.2[-p290]
[ruby-]1.9.2-head
[ruby-]1.9.3[-preview1]
[ruby-]1.9.3-head
ruby-head
...

$ rvm install 1.8.7
...
$ rvm use 1.8.7

Once done, we will create a gemset for Redmine, where RubyGems will install the gems we need in the versions we need, without interfering with the system installation.

$ rvm gemset create redmine
'redmine' gemset created (/Users/olivier/.rvm/gems/ruby-1.8.7-p352@redmine).
$ rvm gemset use redmine

Notes:

  1. The Ruby version and gemset in use are tied to the shell session, so you have to switch every time you start a new shell. It can be done in one step:

    $ rvm use 1.8.7@redmine
    Using /Users/olivier/.rvm/gems/ruby-1.8.7-p352 with gemset redmine
    
  2. The current environment can be queried with rvm-prompt:

    $ rvm-prompt        
    ruby-1.8.7-p352@redmine
    
  3. You can switch back to the system Ruby with:

    $ rvm use system
    Now using system ruby.
    
  4. You do not need sudo to install gems in an RVN gemset, because all the files are contained in your home directory.

  5. Rake commands while using a gemset should be prefixed by rvm:

    $ rvm rake blah blah
    

Installing gems

First of all, let us speed up gem installs 10x, as suggested by The Budding Rubbyist. Add the following to your ~/.gemrc file, creating it if needed:

gem: --no-ri --no-rdoc

That will omit installing the documentation (that we will never read as we’re only interested in using Ruby software, not developing some) when installing gems, a step that is inexplicably very lengthy as it usually takes longer than the “Building native extensions. This could take a while…” step of some gems.

Next, make sure we are using the Ruby and gemset we created earlier:

$ rvm use 1.8.7@redmine

Then install Rails 2.3.14 and i18n 0.4.2 (Rails installs i18n 0.6.0, but remember, Redmine wouldn’t see this one if we forced it into its thick skull with a [we]brick):

gem install rails -v=2.3.14
gem install i18n -v=0.4.2

Note: To figure out that it was Rails 2.3.14 that I wanted, I listed all the versions available and picked the latest 2.3.x:

$ gem query  --all --remote -n '^rails$'

*** REMOTE GEMS ***

rails (3.1.0, 3.0.10, 3.0.9, 3.0.8, 3.0.7, 3.0.6, 3.0.5, 3.0.4, 3.0.3, 3.0.2, 3.0.1, 3.0.0, 2.3.14, 2.3.12, 2.3.11, 2.3.10, 2.3.9, 2.3.8, 2.3.7, 2.3.6, 2.3.5, 2.3.4, 2.3.3, 2.3.2, 2.2.3, 2.2.2, 2.1.2, 2.1.1, 2.1.0, 2.0.5, 2.0.4, 2.0.2, 2.0.1, 2.0.0, 1.2.6, 1.2.5, 1.2.4, 1.2.3, 1.2.2, 1.2.1, 1.2.0, 1.1.6, 1.1.5, 1.1.4, 1.1.3, 1.1.2, 1.1.1, 1.1.0, 1.0.0, 0.14.4, 0.14.3, 0.14.2, 0.14.1, 0.13.1, 0.13.0, 0.12.1, 0.12.0, 0.11.1, 0.11.0, 0.10.1, 0.10.0, 0.9.5, 0.9.4.1, 0.9.4, 0.9.3, 0.9.2, 0.9.1, 0.9.0, 0.8.5, 0.8.0)

The list command is simpler, but it will list everything that starts with rails, which is a bit of a bother:

$ gem list --all --remote rails

*** REMOTE GEMS ***

rails (3.1.0, 3.0.10, 3.0.9, 3.0.8, 3.0.7, 3.0.6, 3.0.5, 3.0.4, 3.0.3, 3.0.2, 3.0.1, 3.0.0, 2.3.14, 2.3.12, 2.3.11, 2.3.10, 2.3.9, 2.3.8, 2.3.7, 2.3.6, 2.3.5, 2.3.4, 2.3.3, 2.3.2, 2.2.3, 2.2.2, 2.1.2, 2.1.1, 2.1.0, 2.0.5, 2.0.4, 2.0.2, 2.0.1, 2.0.0, 1.2.6, 1.2.5, 1.2.4, 1.2.3, 1.2.2, 1.2.1, 1.2.0, 1.1.6, 1.1.5, 1.1.4, 1.1.3, 1.1.2, 1.1.1, 1.1.0, 1.0.0, 0.14.4, 0.14.3, 0.14.2, 0.14.1, 0.13.1, 0.13.0, 0.12.1, 0.12.0, 0.11.1, 0.11.0, 0.10.1, 0.10.0, 0.9.5, 0.9.4.1, 0.9.4, 0.9.3, 0.9.2, 0.9.1, 0.9.0, 0.8.5, 0.8.0)
rails-3-settings (0.1.1)
rails-action-args (0.1.1, 0.1.0)
rails-admin (0.0.0)
rails-and-solid (0.9.1, 0.9.0)
...

I wanted to use SQLite (I couldn’t make the mysql gem build against Zend Server’s MySQL, and I don’t care), so I installed the sqlite3 gem:

gem install sqlite3

We also need to downgrade RubyGems from the latest version (1.8.10 as of writing) to a version that is at least 1.3.7 but not 1.5.0 and not 1.7.0 or later. The version we want can be determined the same way as for Rails, by querying the rubygems-update gem:

$ gem list --all --remote  rubygems-update

*** REMOTE GEMS ***

rubygems-update (1.8.10, 1.8.9, 1.8.8, 1.8.7, 1.8.6, 1.8.5, 1.8.4, 1.8.3, 1.8.2, 1.8.1, 1.8.0, 1.7.2, 1.7.1, 1.7.0, 1.6.2, 1.6.1, 1.6.0, 1.5.3, 1.5.2, 1.5.0, 1.4.2, 1.4.1, 1.4.0, 1.3.7, 1.3.6, 1.3.5, 1.3.4, 1.3.3, 1.3.2, 1.3.1, 1.3.0, 1.2.0, 1.1.1, 1.1.0, 1.0.1, 1.0.0, 0.9.5, 0.9.4, 0.9.3, 0.9.2, 0.9.1, 0.9.0, 0.8.11, 0.8.10, 0.8.8, 0.8.6, 0.8.5, 0.8.4, 0.8.3)

So we want 1.6.2:

$ gem update --system 1.6.2
...
$ gem -v
1.6.2

Note that some older version of the gem command, like the 1.3.7 recommended by Redmine, do not support specifying the version in the gem update --system command. In that case, you would need to get the most recent version and then downgrade it:

$ gem update --system
$ gem update --system 1.6.2

Now let’s proceed to installing Redmine.

Installing Redmine

Download the latest stable release (1.2.1 of 2011-07-11 as of this writing) and unpack it.

We need to disable the built-in Rails 2.3.11 and tell Redmine to use our 2.3.14 version. In config/environment.rb, change these lines:

# Specifies gem version of Rails to use when vendor/rails is not present
RAILS_GEM_VERSION = '2.3.11' unless defined? RAILS_GEM_VERSION

to:

# Specifies gem version of Rails to use when vendor/rails is not present
RAILS_GEM_VERSION = '2.3.14' unless defined? RAILS_GEM_VERSION

Next, move rails out of the vendor directory. If you don’t, Redmine will use the old version no matter what the above line says. I do this:

mv vendor/rails vendor.rails.disabled

By the way, it’s the “read config/environment.rb first” bit in the requirements list and the “when vendor/rails is not present” in the comment above that tell us that we must remove rails from the vendor directory. As I said, a very thorough explanation.

Next, proceed with installation. With SQLite, database.yml looks like this:

production:
  adapter: sqlite3
  database: db/redmine.db

development:
  adapter: sqlite3
  database: db/redmine.db

# Warning: The database defined as "test" will be erased and
# re-generated from your development database when you run "rake".
# Do not set this db to the same as development or production.
test:
  adapter: sqlite3
  database: db/redmine_test.db

Note: The following commands for me were littered with deprecation warnings like this one:

rake/rdoctask is deprecated.  Use rdoc/task instead (in RDoc 2.4.2+)
Please install RDoc 2.4.2+ to generate documentation.
WARNING: 'task :t, arg, :needs => [deps]' is deprecated.  Please use 'task :t, [args] => [deps]' instead.
    at redmine-1.2.1/lib/tasks/email.rake:170

As far as I know they don’t matter, apart from the fact that they render it quite difficult to determine whether the task succeeded or not.

Generate a session store secret (don’t forget to rvm rake instead of just rake):

rvm rake generate_session_store

Create the database structure:

RAILS_ENV=production rvm rake db:migrate

Insert default configuration data in database:

RAILS_ENV=production rvm rake redmine:load_default_data

Then test your installation:

ruby script/server webrick -e production

or

ruby script/server -p 3000 webrick -e production

It takes a little while to start. The first two lines appear quite quickly and let you check that Redmine is running the correct Rails version:

=> Booting WEBrick
=> Rails 2.3.14 application starting on http://0.0.0.0:3000

Then the server is ready when you see this:

=> Call with -d to detach
=> Ctrl-C to shutdown server
[2011-09-25 14:28:09] INFO  WEBrick 1.3.1
[2011-09-25 14:28:09] INFO  ruby 1.8.7 (2011-06-30) [i686-darwin10.8.0]
[2011-09-25 14:28:09] INFO  WEBrick::HTTPServer#start: pid=39670 port=3000

Running Redmine under Unicorn

Disclaimer: I have no idea if using Unicorn to run Redmine is a good idea. I just know that WEBrick is not for production and that Passenger is an Apache module, and I’d rather not bother with it if I can avoid it. After all, we reached thus far without need admin privileges.

I found the instruction at Install Redmine on Ubuntu Server from the Redmine HowTo’s list, except the part about the RVM wrapper that comes from Brandon Tilley.

Install Unicorn, still under RVM:

$ gem install unicorn

Create a wrapper for unicorn_rails. The wrapper will run unicorn_rails with the correct Ruby version and gemset:

$ rvm wrapper ruby-1.8.7-p352@redmine redmine unicorn_rails

This creates a redmine_unicorn_rails command that uses the ruby-1.8.7-p352@redmine environment.

Create the unicorn configuration file unicorn.conf, for example in Redmine’s config directory:

# /var/www/redmine/config/unicorn.conf

# Number of CPU Cores
worker_processes 2

# Same Redmine User
user "redmine", "redmine"
working_directory "/var/www/redmine"

# Listen on Both UNIX and TCP Sockets for Testing
listen "/var/www/redmine/tmp/sockets/.sock", :backlog => 64
listen 8080, :tcp_nopush => true

# Default Timeout
timeout 30

# Pid/Logging Config
pid "/var/www/redmine/tmp/pids/unicorn.pid"
stderr_path "/var/www/redmine/log/unicorn.stderr.log"
stdout_path "/var/www/redmine/log/unicorn.stdout.log"

# Preload Rails App for Performance
preload_app true

before_fork do |server, worker|
  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.connection.disconnect!
end

after_fork do |server, worker|
  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.establish_connection
end

This assumes that you unpacked redmine in /var/www/redmine and that you have a redmine user belonging to a redmine group. If needed, change all occurrences of /var/www/redmine to where you installed Redmine and user "redmine", "redmine" to the user and group you want Redmine to run as. For testing, I used my user account, but it’s certainly not recommended:

user "olivier", "staff"

Then start unicorn using the wrapper, which means we don’t need to enter the RVM environment. This should come in handy if you use launchd or another service system to launch Redmine:

$ sudo -u redmine redmine_unicorn_rails -c /var/www/redmine/config/unicorn.conf -D -E production

If like me you used your own user account for testing purposes, then you don’t need the sudo part:

$ redmine_unicorn_rails -c /var/www/redmine/config/unicorn.conf -D -E production

From the config file, Unicorn runs on port 8080. If you have an Apache web site on port 80 and would like Redmine to appear under a given prefix, and have mod_proxy enabled, then you can probably use a proxy rewrite rule to achieve this.

To stop Unicorn, kill the unicorn_rails master process.

Conclusion

That’s it for today, I hope this helps. As I already mentioned, I am new to Rails applications, so I probably made mistakes. Don’t hesitate to point them out in the comments, I’ll try and update this post accordingly. I might also add something about using launchd to run Redmine under Unicorn and mod_proxy to integrate with an existing web site.

October 8, 2007

Converting a .Mac account to a Family Pack sub-account

.Mac trial accounts and e-mail-only accounts can be converted to a Family Pack sub-account right away without problem under Account → Manage Accounts → Convert an Account, as explained in the .Mac help (trial, email-only). It’s less clear about converting an existing normal account, though.

The key is to turn off automatic renewal (Account → Credit Card Info → Auto-Renew), let the account expire and then convert it (cf. Converting an expired .Mac account to a Family Pack sub-account.) Apparently1, .Mac will keep all your account data for two weeks after expiration, so if you convert it within this period, you won’t lose anything.

1. ‘Apparently’ here meaning “use this information at your own risk.” 

December 7, 2006

Disable Self Heal in Adobe Reader 8

How to prevent Adobe Reader 8 from reinstalling its junk on every launch after you cleaned it up:

  • Open the file Adobe Reader.app/Contents/MacOS/SHInit.xml in a text editor or in Property List Editor;

  • Comment out or delete the following line:

    <key>selfhealingfilename</key><string>RdrENU80SelfHeal.xml</string>
    
  • Save the file;

  • Optional but highly recommended: show Adobe a sign of your triumph.

September 6, 2006

Building a TINI application with Xcode

This post explains how I build applications for the TINI using Xcode. It was written for Xcode 2.4.

  • Create a new Java Tool project in Xcode. The description says “This project builds a library or application as a JAR file.”

  • Double-click the main and only target under Targets. The target editor opens. We will need to do the following:

    • add tiniclasses.jar to the search path for Java classes;
    • tell javac to target version 1.1 of the virtual machine;
    • tell Xcode to generate a class hierarchy instead of a Java archive (JAR file).
  • In Search Paths, drag and drop the tiniclasses.jar file (from the TINI SDK) under Java Classes, or select Java Classes, click the + button and select the tiniclasses.jar file in the file dialogue. This adds the TINI classes to the class path.

  • In Java Compiler Settings, change Target VM Version to 1.1 and Source Version to 1.3. The TINI runs a 1.1 virtual machine and a source version of 1.3 seems to be required in order to compile version 1.1 classes.

  • In Java Archive Settings, change Product Type to Class Hierarchy. This will create a folder containing the class files, on which we can subsequently run the TINIConvertor tool, instead of a JAR file.

The final step is to call the TINIConvertor tool through a shel script build phase, in order to build the TINI executable:

  • Right-click on the last build phase in the left pane of the target editor (most likely Copy Files) and select New Build Phase — New Shell Script Build Phase. A Run Script build phase will be added to the end of the build phase list.

  • In the Script text field of the new build phase, paste the following script:

    TINIPATH=$LOCAL_LIBRARY_DIR/TINI/bin
     
    java -classpath "$TINIPATH/tini.jar" TINIConvertor \
        -f "$CLASS_FILE_DIR" -d "$TINIPATH/tini.db" \
        -o "$TARGET_BUILD_DIR/$PRODUCT_NAME.tini"
    

    Replace the value for TINIPATH with the actual path to your copy of the TINI SDK, i.e. the folder that contains tini.jar and tini.db. We use $CLASS_FILE_DIR to locate the class files, $TARGET_BUILD_DIR to locate the build directory and $PRODUCT_NAME to name the executable.

That’s it. Unfortunately, it seems that you have to go through these steps for both the Debug and Release build configurations. You can now build your project and if all goes well, a .tini file should land in your build/Debug or build/Release folder. If not, drop me a note.

May 27, 2006

How iTunes selects the storefront

Following the itmshack: Grab Album Description Text from the iTunes Music Store article on the O’Reilly Mac DevCenter Blog, and on a Friend’s suggestion, and out of personal interest, I tried to figure out how iTunes specifies the storefront (or country) when talking to the iTunes Music Store. Running tcpflow while visiting the Swiss French store gave it away quite quickly:

010.000.000.003.52678-080.067.084.016.00080: GET /WebObjects/MZStore.woa/wa/viewAlbum?id=154148337&s=143459 HTTP/1.1
X-Apple-Tz: 7200
X-Apple-Store-Front: 143459-2
Referer: http://ax.phobos.apple.com.edgesuite.net/WebObjects/MZStore.woa/wa/storeFront
User-Agent: iTunes/6.0.4 (Macintosh; U; PPC Mac OS X 10.4.6)
Accept-Language: en-us, en;q=0.50
X-Apple-Validation: 661D3B2F-DB90D338EA5BA65F951043499F6627ED
Accept-Encoding: gzip, x-aes-cbc
Connection: close
Host: ax.phobos.apple.com.edgesuite.net

The storefront is selected with the X-Apple-Store-Front HTTP request header. The number can be extracted by Control-clicking an item in iTunes and choosing ‘Copy iTunes Music Store URL’. As we can see, the s query parameter in the URL gives the storefront number:

http://phobos.apple.com/WebObjects/MZStore.woa/wa/viewAlbum?id=154148337&s=143459

The additional -2 in the HTTP header selects the language. If it’s omitted, the default language is used (German for the Swiss French store). Here are the values I could identify before I ran out of patience:

1: English
2: English (slightly different texts)
3: French
4: German
5: French (slightly different texts)

You can easily tell curl to add the X-Apple-Store-Front header:

curl --header 'X-Apple-Store-Front: 143459-2' -s 'http://ax.phobos.apple.com.edgesuite.net/WebObjects/MZStore.woa/wa/viewAlbum?id=154148337' | gunzip

Modification of the itmshack script is left as an exercise for the reader (or for the script’s author ;-)).

June 21, 2005

Freezing the play order of a shuffled iTunes playlist

The problem with shuffled playlists in iTunes is that if I quit iTunes at the end of the day, relaunch it the next day and want to resume playing the same playlist where I left it, iTunes will shuffle it again, meaning I'm likely to hear songs I already heard the day before. On the other hand, if I turn shuffle off, it will sort the songs in the order I put them when I created the playlist, which is likely to be in Artist/Album order since this is how I browse my library when making a playlist.

The solution is simple but not well documented, if at all. After making the playlist, turn shuffle on. Then Control-click in the song list and choose "Copy To Play Order." It will change the 'natural' order of the playlist to the current sequence. Then turn shuffle back off and the order will be retained.

October 29, 2004

Using wget to download a URL stored in the clipboard

Here is an alias used to quickly download a link using wget:

alias cget 'pbpaste | wget -i -'

pbpaste is a Mac OS X only command line utility that prints the clipboard to the standard output. wget’s -i option tells it to read URL(s) from a file, and - makes it use the standard input in place of a file.

To use it, copy a URL, open a new Terminal window and type cget. Download should begin, with all of wget’s great features like perseverence and robustness.

This technique will be improved in an upcoming article has been used by yours truly for almost a year and a half without feeling the need for improvement.

August 24, 2004

Manipulating the clipboard in the Terminal: pbcopy and pbpaste

Mac OS X provides two utilities that make it possible to manipulate the clipboard in the Terminal: pbcopy and pbpaste.

pbcopy puts the data it receives via standard input into the clipboard.

echo "Hello, World!" | pbcopy
pbcopy < aFile

pbpaste does the opposite. It ouputs the clipboard to the standard output.

pbpaste
pbpaste > anotherFile

It is possible to select the preferred format for the clipboard data using the -Prefer option:

-Prefer rtf|ps|ascii

Both commands also accept the -help option:

Usage: pbcopy [-help]
Usage: pbpaste [-help] [-Prefer rtf|ps|ascii]

August 14, 2004

Nailed it down

Xcode has been driving me mad recently. I am writing TINI applications in Java, which requires me to feed the class files to a converter application. I had a little trouble figuring out how to make Xcode generate and run on a class hierarchy instead of a jar file but I will talk about it in details in an upcoming entry. The topic today is the bug I discovered (and fixed!) in Xcode when working with class hierarchies.

Since I already wrote about it several times for the Java Dev and Xcode Users mailing lists, I will limit myself to posting the report I submitted to Apple.

1. Configuration

  • Mac OS X 10.3.5
  • Xcode 1.5
  • Java 1.4.2 Update 1
  • Java 1.4.2 Update 1 Developer Tools

2. Summary

When configuring a Java Tool project to generate a class hierarchy instead of a Java archive, the class files are copied into TARGET_BUILD_DIR/PRODUCT_NAME (i.e. PRODUCT_CLASS_FILE_DIR) the first time the project is built. However, on subsequent builds, the classes in that folder are not updated, even when the sources are. Therefore, the project runs on old class files and so do potential shell script build phases that would post process the class files.

3. Steps to reproduce

  • Create a new Java Tool project in Xcode;
  • In the active target, change the Java Archive Settings / Product type to Class Hierarchy (or in expert mode, set JAVA_ARCHIVE_CLASSES = NO);
  • Build the project;
  • In the Finder, take note the modification date of the class file in build/PRODUCT_NAME;
  • Make a modification to the sources, e.g. change "Hello World!" to "Take over the World!";
  • Save and build the project;
  • In the Finder, notice that the class file has not been updated since the modification date did not change.

4. Expected results

Every time the project is built, the class hierarchy should be copied from CLASS_FILE_DIR to PRODUCT_CLASS_FILE_DIR if it has been modified, as it happens the first time the project is built.

5. Actual results

The class hierarchy is only copied on the first build or after a target clean-up.

6. Cause

Watching the build process from a shell using xcodebuild instead of the Xcode GUI shows that on a clean target, after the Java Archive Files build phase, the PRODUCT_CLASS_FILE_DIR is created and ditto is used to copy the class files from CLASS_FILE_DIR to PRODUCT_CLASS_FILE_DIR:

BuildPhase <JavaArchiveFiles>Hello
    echo Completed phase "<JavaArchiveFiles>" for "<JavaArchiveFiles>Hello"
Completed phase <JavaArchiveFiles> for <JavaArchiveFiles>Hello

Mkdir […]/Hello/build/Hello 
    /bin/mkdir  -p […]/Hello/build/Hello 

Ditto […]/Hello/build/Hello 
    /usr/bin/ditto  […]/Hello/build/Hello.build/Hello.build\
    /JavaClasses […]/Hello/build/Hello

However, when PRODUCT_CLASS_FILE_DIR exists, neither of these two operations is executed.

7. Workaround

Cleaning the active target prior to building or deleting the PRODUCT_CLASS_FILE_DIR folder causes Xcode to create it again, with up-to-date files. However, emptying this folder is not enough.

8. Resolution

The build instructions are defined in

/Developer/Makefiles/pbx_jamfiles/ProjectBuilderJambase.

The parts of interest are the definition of Mkdir (ProjectBuilderJambase, line 299):

# Mkdir <directory>
# Creates <directory>
rule Mkdir
{
    # Only existence of the directory matters
    NOUPDATE $(1) ;
}
actions together piecemeal Mkdir
{
    $(MKDIR) -p $(1:Q)
}

and the section where the class files are copied using ditto, where it looks like someone has been bothered by this bug for some time (line 4267):

if $(JAVA_ARCHIVE_CLASSES) != YES {
    # !!!:cmolick:20020123 product class file dir not always made?!
    Mkdir $(PRODUCT_CLASS_FILE_DIR) ;
    ProductFile $(PRODUCT_CLASS_FILE_DIR) ;
    Ditto $(PRODUCT_CLASS_FILE_DIR) : $(CLASS_FILE_DIR) ;
if $(MERGED_ARCHIVES) {
        DEPENDS $(PRODUCT_CLASS_FILE_DIR) : $(MERGED_ARCHIVES) ;
    }
    else {
        DEPENDS $(PRODUCT_CLASS_FILE_DIR) : $(JAVA_COMPILE_TARGET) ;
    }
}

It is actually the Mkdir command that prevents the copy to execute, even though mkdir -p exits 0 regardless of whether the directory to create is already present. I have no idea whatsoever of the language used to write ProjectBuilderJambase but I can make a pretty safe guess that the culprit is the NOUPDATE statement in the Mkdir rule. Indeed, removing it solves the problem, albeit not in a very clean way as this statement is probably necessary elsewhere (although it might be problematic elsewhere as well).

The fix I implemented was to create a new action, ForceMkDir, to use in this particular case (ProjectBuilderJambase, line 310):

# ForceMkdir <directory>
# Creates <directory>, even if it already exists.
#
# This modification is for the Ditto step that copies class files in
# Java projects (line 4267-78). The original Mkdir action above seems
# to prevent ditto from executing when the directory already exists.
#
# Corrected 2004-08-12 by Ölbaum, who's quite proud of having fixed a
# glitch in a program written in a language he does not even know the
# name of.
actions together piecemeal ForceMkdir
{
    $(MKDIR) -p $(1:Q)
}

and (line 4269):

# !!!!:Ölbaum:20040812 fixed!
ForceMkdir $(PRODUCT_CLASS_FILE_DIR) ;

9. Enclosures

A patch file for these modifications: ProjectBuilderJambase.diff.

10. Conclusion

After applying these modifications, the class hierarchy in PRODUCT_CLASS_FILE_DIR is properly updated each time the classes are recompiled.

August 13, 2004

Pipes and redirections in sh

A quick reference to pipes and redirections in sh.

Notes

In sh, the standard output is identified as file descriptor 1 and the diagnostic output as file descriptor 2.

Pipes

  • Use the standard output of program_1 as the standard input of program_2:

    program_1 | program_2
    
  • Use the standard and diagnostic outputs of program_1 as the standard input of program_2:

    program_1 2>&1 | program_2
    

Redirections

  • Open file name as the standard input:

    program < name
    
  • Use file name as the standard output:

    program > name
    
  • Append the output to the end of file name:

    program >> name
    
  • Route both the standard and diagnostic outputs to file name:

    ( program 2>&1 ) > name
    
  • Append both the standard and diagnostic outputs to file name:

    ( program 2>&1 ) >> name
    

Tips

  • To route the standard and diagnostic outputs to separate files, use the following syntax:

    program > outfile 2> errfile
    
  • To route the diagnostic output to a file and the standard output to the terminal, use:

    program 2> errfile
    
  • To retain only the diagnostic output, redirect the standard output to /dev/null:

    program > /dev/null
    program > /dev/null 2> errfile
    
  • In order to test pipes and redirections, use the following Perl program:

    #!/usr/bin/perl
    
    
    print STDOUT "STDOUT: Standard output\n";
    print STDERR "STDERR: Diagnostic output\n";
    

August 9, 2004

Pipes and redirections in tcsh

A quick reference to pipes and redirections in tcsh.

Pipes

  • Use the standard output of program_1 as the standard input of program_2:

    program_1 | program_2
    
  • Use the standard and diagnostic outputs of program_1 as the standard input of program_2:

    program_1 |& program_2
    

Redirections

  • Open file name as the standard input:

    program < name
    
  • Use file name as the standard output:

    program > name
    
  • Append the output to the end of file name:

    program >> name
    
  • Route both the standard and diagnostic outputs to file name:

    program >& name
    
  • Append both the standard and diagnostic outputs to file name:

    program >>& name
    

Tips

  • To route the standard and diagnostic outputs to separate files, use the following syntax:

    ( program > outfile ) >& errfile
    
  • To route the diagnostic output to a file and the standard output to the terminal, use:

    ( program > /dev/tty ) >& errfile
    
  • To retain only the diagnostic output, redirect the standard output to /dev/null:

    program > /dev/null
    ( program > /dev/null ) >& errfile
    
  • In order to test pipes and redirections, use the following Perl program:

    #!/usr/bin/perl
    
    
    print STDOUT "STDOUT: Standard output\n";
    print STDERR "STDERR: Diagnostic output\n";
    

Knowledge base migration

I have decided to move my knowledge base to my main blog. The benefits are threefold:

  • I can repost knowledge base articles and give the impression that I update my blog more often;
  • Since I post other things here, it becomes less visible that I very rarely update the knowledge base;
  • I save one blog for when I upgrade to Movable Type 3

I will start moving the articles right now, starting with the most popular one: Pipes and redirections in tcsh. Of course, the old knowledge base permalinks will still work, long live mod_rewrite.

Do not meddle in the affairs of Coding Ninjas, for they are subtle and quick to anger.