As soon as the Pi Zero came out, I started on thinking about Clusters again. I wanted to create an big - but at the same time, cheap cluster.
Yes, an Pi Zero is not nearly as fast, as an RPi 2. And yes, there are some problems with this idea - especially about the fact, that the Pi Zero is not as common as an - say - RPi Modell B v. 2.0 - but - as this is more about science and trying to just "do it" - I tried it anyway.

TLDR; Yes, it works - and better than you might think :)!

So, the basic problem about the Pi Zero are its interfaces: Yes, we got USB, but none Ethernet Port. So the basic approach would be to buy an 5€ Pi as well as an about 8€ AX88772 Ethernet Interface and some USB OTG Adapters - an we would end up with about 15€+ for each member of the cluster. Well, that is reasonable - but bulky and "not sexy".

0. Building mpich
I used some old Appliance Image I created from an Minibian Wheezy Image (https://minibianpi.wordpress.com/) earlier this year - for the 1.) section on the RPi Modell B pre 2.0 and RPi Modell A+. For the 2.) section, I used an special Appliance Image I made from an Minibian Jessie Image. However, I will document needed changes here, to get it running from any source. I recommend the Minibian Jessie Image as starting point, with this changes:


apt-get update
apt-get install -y raspi-config keyboard-configuration
raspi-config
# Default Configuration and Expand Filesystem using raspi-config
# Enter Finish and press Yes on Reboot the Device

apt-get install -y rpi-update sudo
apt-get -y dist-upgrade
reboot

rpi-update
reboot

# Create Default User pi
adduser pi
# Enter Password as wanted, i.e. raspberry
# Add user to default groups
usermod -a -G pi,adm,dialout,cdrom,audio,video,plugdev,games,users pi
# Add sbin Paths to pi
echo 'export PATH="$PATH:/sbin:/usr/sbin:usr/local/sbin"' >> /home/pi/.bashrc
# Add user to sudo
visudo
# Add under
# # User privilege specification
# root ALL=(ALL:ALL) ALL
pi ALL=(ALL:ALL) ALL
# Save and Exit
reboot

# Disable root login
sudo passwd -l root


or - and default RPi Jessie Image.

Building MPICH 3 was quite easy:


# Update and Install Dependencies, then reboot
sudo apt-get update
sudo apt-get -y dist-upgrade
sudo apt-get install -y build-essential
sudo reboot

# Make MPICH 3.2
cd ~
wget http://www.mpich.org/static/downloads/3.2/mpich-3.2.tar.gz
tar -xvzf mpich-3.2.tar.gz
cd mpich-3.2
# This will take some time
sudo ./configure --disable-fortran
# This will take several cups of tea ;)
sudo make
sudo make install


# Create SSH on Master, distribute to Slaves

cd ~
ssh-keygen -t rsa –C "raspberrypi"

Default location should be set to /home/pi/.ssh/id_rsa if you're using the standard user pi. Then choose this command to distribute the key to all your "slave maschines":
cat ~/.ssh/id_rsa.pub | ssh pi@IP_OF_SLAVES "mkdir .ssh;cat >> .ssh/authorized_keys"
( Was taken from http://www.southampton.ac.uk/~sjc/raspberrypi/ - he was the original father of the RPi Clusters and his work inspired me already years ago - so please read and support his work :)! Additional infos can be found at http://westcoastlabs.blogspot.co.uk/2012/06/parallel-processing-on-pi-bramble.html)
You could also just
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
to your own authorized files, shutdown your Master Pi after that and clone the card several times for all your Clients. This way, you would only need to do the work once - however, maybe you should release the keys in ~/.ssh/ so that only your Master Pi could command the slaves

1. Frist Try: Serial (did work, but not choosen)

First idea was, to use the Serial Line of the Pi Zero for IP communication:
I wanted to have an Master Pi (Modell B) with Ethernet Port for network connectivity and connect up to 4 Pi Zero to it via Serial. And as the RPis only have one serial port, I would use an Serial to SPI Converter to keep it small and simple. But as it turns out, I could not find the MAX14830 (https://www.maximintegrated.com/en/products/interface/controllers-expanders/MAX14830.html) for sale on the net, so I got two 2 Port MAX3109 Serial to SPI Converters (https://www.maximintegrated.com/en/products/interface/controllers-expanders/MAX3109.html).
They were on the way to my home, so I wanted to already test some basic stuff by using an RPi Modell B pre 2.0 and an RPi Modell A+.

As we only had one serial port, we could only drive the one RPi Modell A+ with that. I will use the B pre 2.0 as Master, and the A+ as Slave. First, we connected both RPis via Serial (both shutdown and unpluged!). Just connect the Serial TX of RPi B to the Serial RX of the RPi A+ and vice versa. Then connect a Ground Pin of the RPi B to the RPi A+. Thats it.
Then we powered on the RPis and made some changes:
(Some ideas taken from MagPi 41: https://www.raspberrypi.org/magpi/)

Guest:

# I actually prepared the whole sdcard of the RPi A+ Guest while having that SD Card in the RPi B - because of the networking connection :).
# Install ppp
sudo apt-get install ppp -y
# Change Hostname
sudo vi /etc/hostname
sudo vi /etc/hosts
# Add DNS Server
echo 'nameserver 8.8.8.8' | sudo tee --append /etc/resolv.conf
# Remove Serial Console
cd /boot
sudo cp cmdline.txt old_cmdline
sudo vi cmdline.txt
# Normal should read about
dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsc
# Remove console=ttyAMA0,115200 from that line, save and quit

# Disable Serial Console by using sudo raspi-config and the option, to be sure

# Increase Clock on the Serial Line, so drive the Serial Line at not only 115200 baud, but 1 MBit/s (taken from: http://www.thedevilonholiday.co.uk/raspberry-pi-increase-uart-speed/)
echo 'init_uart_clock=64000000' | sudo tee --append /boot/config.txt

# Add the following line to /etc/rc.local before exit 0
pppd /dev/ttyAMA0 1000000 10.0.5.2:10.0.5.1 noauth local defaultroute nocrtscts xonxoff persist maxfail 0 holdoff 1 &

# and shutdown
sudo shutdown -h now

After that, insert that SD Card into the A+.

Host:

# Now insert the real SD Card for the B into the B and power it on.
# Install ppp
sudo apt-get install ppp -y
# Enable ipv4 Forward for networking access
echo 'net.ipv4.ip_forward=1' | sudo tee --append /etc/sysctl.conf
# Remove Serial Console
cd /boot
sudo cp cmdline.txt old_cmdline
sudo vi cmdline.txt
# Normal should read about
dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsc
# Remove console=ttyAMA0,115200 from that line, save and quit

# Disable Serial Console by using sudo raspi-config and the option, to be sure

# Increase Clock on the Serial Line, so drive the Serial Line at not only 115200 baud, but 1 MBit/s (taken from: http://www.thedevilonholiday.co.uk/raspberry-pi-increase-uart-speed/)
echo 'init_uart_clock=64000000' | sudo tee --append /boot/config.txt

# Reboot
reboot

# After having rebooted the RPi B, boot up the RPi A+ as well.
# Wait a little bit, then...

# Start PPP Connection
sudo pppd /dev/ttyAMA0 1000000 10.0.5.1:10.0.5.2 proxyarp local noauth nodetach nocrtscts xonxoff passive persist maxfail 0 holdoff 1


In a new putty window, you can now ping 10.0.5.2 - your RPi A+ and can SSH into it.

After that, I could use MPI with both machines, B and A+ - by entering their IP addresses into the machinefile and executing the cpi example to crunch some Pi Numbers.

But after all - it was quite ineffective and slow. So I tried to think of something better.. And then LadyAda came with her christmas present to me:
https://learn.adafruit.com/turning-your-raspberry-pi-zero-into-a-usb-gadget/ethernet-gadget - that was the moment my jaw dropped and I thought "YES! Thats it!".

2. Second Try: PiZero on Virtual Ethernet (Solution)

Now my prefered solution: As the USB Port of the PiZero is an real OTG Port, you could repurpose it as an Serial or even Virtual Ethernet port. And hell, Lady Ada striked again :)! So to sum it up shortly:
I build my MPICH as mentioned in 0) on an Minibian Jessie image (SDCard running on an RPi B). Then I installed her special kernel:


cd ~
wget http://adafruit-download.s3.amazonaws.com/gadgetmodulekernel_151224a.tgz -o gadgetkernel.tgz
tar -xvzf gadgetkernel.tgz

sudo mv /boot/kernel.img /boot/kernelbackup.img
sudo mv tmp/boot/kernel.img /boot

sudo mv tmp/boot/overlays/* /boot/overlays
sudo mv tmp/boot/*dtb /boot
sudo cp -R tmp/boot/modules/lib/* /lib

# Load virtual ethernet module on boot
echo 'g_ether' | sudo tee --append /etc/modules

# Add settings to network interfaces
echo '
allow-hotplug usb0
iface usb0 inet static
address 192.168.7.2
netmask 255.255.255.0
network 192.168.7.0
broadcast 192.168.7.255
gateway 192.168.7.1' | sudo tee --append /etc/network/interfaces

# Add DNS Server
echo 'nameserver 8.8.8.8' | sudo tee --append /etc/resolv.conf

# Some additional tweaks:
Add

# Turn HDMI Off
/usr/bin/tvservice -o
# Turn HDMI Back On
#/usr/bin/tvservice -p

# Turn ACT LED Off on Pi Zero
echo none | sudo tee /sys/class/leds/led0/trigger
echo 1 | sudo tee /sys/class/leds/led0/brightness


to your /etc/rc.local before exit 0 to turn off the HDMI Interface on boot,
as well as the LED of the Pi Zero to use less energy. Found on:
http://www.midwesternmac.com/blogs/jeff-geerling/raspberry-pi-zero-conserve-energy and http://www.midwesternmac.com/blogs/jeff-geerling/controlling-pwr-act-leds-raspberry-pi

This was enough to create an Pi Zero Slave Image.
Shutdown the RPi now with

sudo shutdown -h now

remove the Power and insert the SDcard into your Pi Zero.

On the Master Machine, I did following changes:

# Add settings to network interfaces
echo '
allow-hotplug usb0
iface usb0 inet static
address 192.168.7.1
netmask 255.255.255.0
network 192.168.7.0
broadcast 192.168.7.255' | sudo tee --append /etc/network/interfaces

# Allow Ipv4 Forward
echo 'net.ipv4.ip_forward=1' | sudo tee --append /etc/sysctl.conf

# Install iptables
sudo apt-get install -y iptables

# Define NATing rules
sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
sudo iptables -A FORWARD -i eth0 -o usb0 -m state --state RELATED,ESTABLISHED -j ACCEPT
sudo iptables -A FORWARD -i usb0 -o eth0 -j ACCEPT

# Save NAT rules / load iptables on interface up
sudo touch /etc/iptables_masq.rules
sudo chown pi:pi /etc/iptables_masq.rules
sudo iptables-save > /etc/iptables_masq.rules

Add
pre-up iptables-restore < /etc/iptables_masq.rules under the eth0 section in the network interfaces sudo vi /etc/network/interfaces i.e. auto eth0 iface eth0 inet dhcp pre-up iptables-restore < /etc/iptables_masq.rules ( Info taken from: http://serverfault.com/questions/405628/routing-traffic-on-ubuntu-to-give-raspberry-pi-internet-access )

#After that, I shutdown the RPi via
sudo shutdown -h now
#removed power from it.


Then I attached the Pi Zero to the Hub of Pi B via an Micro USB Cable by using the Micro USB OTG Slot on the Pi Zero, connecting it to the Hub of the Pi Modell B. Next, I powered up the Pi B - and both booted.

I pinged 192.168.7.2 - which was the IP of the Pi Zero - and it answered. Now I only had to use cat ~/.ssh/id_rsa.pub | ssh pi@192.168.7.2 "mkdir .ssh;cat >> .ssh/authorized_keys" from Section 0 to get the SSH Key, created in Step 0 onto the Pi Zero and could use that to automatically login in into the Pi Zero.
With the new IP of the RPi B and Pi Zero in the machinefile of mpich I could then use my both RPis with higher speed and nearly zero costs for cabeling and power :)!
The clou: I don't need an additional powersupply for the Pi Zero - nor network adapters, RJ45 cabling, an switch - only one USB A to USB Micro cable per Pi Zero - and maybe an big, active USB Hub ;)!

Now, I need to get more Pi Zeros - I plan on using an Modell B as Master with an big active USB Hub to support 4 Pi Zeros - or an Modell B+ with an REALLY BEFFY USB Supply to work them all the same RPi - but that would come down to trying this... And I got only one Pi Zero - so I need some more time (or some sponsors?) to get me more RPi Zeros to try and see, whether this approach does scale ;)!

Best thing: This can also be used to try the awesome work of http://blog.hypriot.com/ to build an Docker Cluster from that - cool, ain't it?

Merry Christmas :)!

Yours,

Nico