{"id":1151,"date":"2015-12-26T21:13:03","date_gmt":"2015-12-26T20:13:03","guid":{"rendered":"https:\/\/www.nico-maas.de\/?p=1151"},"modified":"2015-12-26T22:25:06","modified_gmt":"2015-12-26T21:25:06","slug":"rpi-the-cheapest-raspberry-pi-cluster-ever-made","status":"publish","type":"post","link":"https:\/\/www.nico-maas.de\/?p=1151","title":{"rendered":"[RPi] The cheapest Raspberry Pi Cluster Ever Made"},"content":{"rendered":"

As soon as the Pi Zero came out, I started on thinking about Clusters again. I wanted to create an big - but at the same time, cheap cluster.
\nYes, an Pi Zero is not nearly as fast, as an RPi 2. And yes, there are some problems with this idea - especially about the fact, that the Pi Zero is not as common as an - say - RPi Modell B v. 2.0 - but - as this is more about science and trying to just \"do it\" - I tried it anyway.<\/p>\n

TLDR; Yes, it works - and better than you might think :)!<\/p>\n

So, the basic problem about the Pi Zero are its interfaces: Yes, we got USB, but none Ethernet Port. So the basic approach would be to buy an 5\u20ac Pi as well as an about 8\u20ac AX88772 Ethernet Interface and some USB OTG Adapters - an we would end up with about 15\u20ac+ for each member of the cluster. Well, that is reasonable - but bulky and \"not sexy\".<\/p>\n

0. Building mpich
\nI used some old Appliance Image I created from an Minibian Wheezy Image (https:\/\/minibianpi.wordpress.com\/<\/a>) earlier this year - for the 1.) section on the RPi Modell B pre 2.0 and RPi Modell A+. For the 2.) section, I used an special Appliance Image I made from an Minibian Jessie Image. However, I will document needed changes here, to get it running from any source. I recommend the Minibian Jessie Image as starting point, with this changes:<\/p>\n


\napt-get update
\napt-get install -y raspi-config keyboard-configuration
\nraspi-config
\n# Default Configuration and Expand Filesystem using raspi-config
\n# Enter Finish and press Yes on Reboot the Device<\/p>\n

apt-get install -y rpi-update sudo
\napt-get -y dist-upgrade
\nreboot<\/p>\n

rpi-update
\nreboot<\/p>\n

# Create Default User pi
\nadduser pi
\n# Enter Password as wanted, i.e. raspberry
\n# Add user to default groups
\nusermod -a -G pi,adm,dialout,cdrom,audio,video,plugdev,games,users pi
\n# Add sbin Paths to pi
\necho 'export PATH=\"$PATH:\/sbin:\/usr\/sbin:usr\/local\/sbin\"' >> \/home\/pi\/.bashrc
\n# Add user to sudo
\nvisudo
\n# Add under
\n# # User privilege specification
\n# root ALL=(ALL:ALL) ALL
\npi ALL=(ALL:ALL) ALL
\n# Save and Exit
\nreboot<\/p>\n

# Disable root login
\nsudo passwd -l root
\n<\/code><\/p>\n

or - and default RPi Jessie Image.<\/p>\n

Building MPICH 3 was quite easy:<\/p>\n


\n# Update and Install Dependencies, then reboot
\nsudo apt-get update
\nsudo apt-get -y dist-upgrade
\nsudo apt-get install -y build-essential
\nsudo reboot<\/p>\n

# Make MPICH 3.2
\ncd ~
\nwget http:\/\/www.mpich.org\/static\/downloads\/3.2\/mpich-3.2.tar.gz
\ntar -xvzf mpich-3.2.tar.gz
\ncd mpich-3.2
\n# This will take some time
\nsudo .\/configure --disable-fortran
\n# This will take several cups of tea ;)
\nsudo make
\nsudo make install
\n<\/code><\/p>\n

# Create SSH on Master, distribute to Slaves
\n
\ncd ~
\nssh-keygen -t rsa \u2013C \"raspberrypi\"
\n<\/code>
\nDefault location should be set to \/home\/pi\/.ssh\/id_rsa if you're using the standard user pi. Then choose this command to distribute the key to all your \"slave maschines\":
\ncat ~\/.ssh\/id_rsa.pub | ssh pi@IP_OF_SLAVES \"mkdir .ssh;cat >> .ssh\/authorized_keys\"
\n( Was taken from http:\/\/www.southampton.ac.uk\/~sjc\/raspberrypi\/<\/a> - he was the original father of the RPi Clusters and his work inspired me already years ago - so please read and support his work :)! Additional infos can be found at http:\/\/westcoastlabs.blogspot.co.uk\/2012\/06\/parallel-processing-on-pi-bramble.html<\/a>)
\nYou could also just
\ncat ~\/.ssh\/id_rsa.pub >> ~\/.ssh\/authorized_keys
\nto your own authorized files, shutdown your Master Pi after that and clone the card several times for all your Clients. This way, you would only need to do the work once - however, maybe you should release the keys in ~\/.ssh\/ so that only your Master Pi could command the slaves<\/p>\n

1. Frist Try: Serial (did work, but not choosen)<\/p>\n

\"ppplink\"<\/a><\/p>\n

First idea was, to use the Serial Line of the Pi Zero for IP communication:
\nI wanted to have an Master Pi (Modell B) with Ethernet Port for network connectivity and connect up to 4 Pi Zero to it via Serial. And as the RPis only have one serial port, I would use an Serial to SPI Converter to keep it small and simple. But as it turns out, I could not find the MAX14830 (https:\/\/www.maximintegrated.com\/en\/products\/interface\/controllers-expanders\/MAX14830.html<\/a>) for sale on the net, so I got two 2 Port MAX3109 Serial to SPI Converters (https:\/\/www.maximintegrated.com\/en\/products\/interface\/controllers-expanders\/MAX3109.html<\/a>).
\nThey were on the way to my home, so I wanted to already test some basic stuff by using an RPi Modell B pre 2.0 and an RPi Modell A+.<\/p>\n

As we only had one serial port, we could only drive the one RPi Modell A+ with that. I will use the B pre 2.0 as Master, and the A+ as Slave. First, we connected both RPis via Serial (both shutdown and unpluged!). Just connect the Serial TX of RPi B to the Serial RX of the RPi A+ and vice versa. Then connect a Ground Pin of the RPi B to the RPi A+. Thats it.
\nThen we powered on the RPis and made some changes:
\n(Some ideas taken from MagPi 41: https:\/\/www.raspberrypi.org\/magpi\/<\/a>)<\/p>\n

Guest:
\n
\n# I actually prepared the whole sdcard of the RPi A+ Guest while having that SD Card in the RPi B - because of the networking connection :).
\n# Install ppp
\nsudo apt-get install ppp -y
\n# Change Hostname
\nsudo vi \/etc\/hostname
\nsudo vi \/etc\/hosts
\n# Add DNS Server
\necho 'nameserver 8.8.8.8' | sudo tee --append \/etc\/resolv.conf
\n# Remove Serial Console
\ncd \/boot
\nsudo cp cmdline.txt old_cmdline
\nsudo vi cmdline.txt
\n# Normal should read about
\ndwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1 root=\/dev\/mmcblk0p2 rootfstype=ext4 elevator=deadline fsc
\n# Remove console=ttyAMA0,115200 from that line, save and quit<\/p>\n

# Disable Serial Console by using sudo raspi-config and the option, to be sure<\/p>\n

# Increase Clock on the Serial Line, so drive the Serial Line at not only 115200 baud, but 1 MBit\/s (taken from: http:\/\/www.thedevilonholiday.co.uk\/raspberry-pi-increase-uart-speed\/<\/a>)
\necho 'init_uart_clock=64000000' | sudo tee --append \/boot\/config.txt<\/p>\n

# Add the following line to \/etc\/rc.local before exit 0
\npppd \/dev\/ttyAMA0 1000000 10.0.5.2:10.0.5.1 noauth local defaultroute nocrtscts xonxoff persist maxfail 0 holdoff 1 &<\/p>\n

# and shutdown
\nsudo shutdown -h now
\n<\/code>
\nAfter that, insert that SD Card into the A+.<\/p>\n

Host:
\n
\n# Now insert the real SD Card for the B into the B and power it on.
\n# Install ppp
\nsudo apt-get install ppp -y
\n# Enable ipv4 Forward for networking access
\necho 'net.ipv4.ip_forward=1' | sudo tee --append \/etc\/sysctl.conf
\n# Remove Serial Console
\ncd \/boot
\nsudo cp cmdline.txt old_cmdline
\nsudo vi cmdline.txt
\n# Normal should read about
\ndwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1 root=\/dev\/mmcblk0p2 rootfstype=ext4 elevator=deadline fsc
\n# Remove console=ttyAMA0,115200 from that line, save and quit<\/p>\n

# Disable Serial Console by using sudo raspi-config and the option, to be sure<\/p>\n

# Increase Clock on the Serial Line, so drive the Serial Line at not only 115200 baud, but 1 MBit\/s (taken from: http:\/\/www.thedevilonholiday.co.uk\/raspberry-pi-increase-uart-speed\/<\/a>)
\necho 'init_uart_clock=64000000' | sudo tee --append \/boot\/config.txt<\/p>\n

# Reboot
\nreboot<\/p>\n

# After having rebooted the RPi B, boot up the RPi A+ as well.
\n# Wait a little bit, then...<\/p>\n

# Start PPP Connection
\nsudo pppd \/dev\/ttyAMA0 1000000 10.0.5.1:10.0.5.2 proxyarp local noauth nodetach nocrtscts xonxoff passive persist maxfail 0 holdoff 1
\n<\/code><\/p>\n

In a new putty window, you can now ping 10.0.5.2 - your RPi A+ and can SSH into it.<\/p>\n

After that, I could use MPI with both machines, B and A+ - by entering their IP addresses into the machinefile and executing the cpi example to crunch some Pi Numbers.<\/p>\n

But after all - it was quite ineffective and slow. So I tried to think of something better.. And then LadyAda came with her christmas present to me:
\nhttps:\/\/learn.adafruit.com\/turning-your-raspberry-pi-zero-into-a-usb-gadget\/ethernet-gadget<\/a> - that was the moment my jaw dropped and I thought \"YES! Thats it!\".<\/p>\n

2. Second Try: PiZero on Virtual Ethernet (Solution)<\/p>\n

\"usblink\"<\/a><\/p>\n

Now my prefered solution: As the USB Port of the PiZero is an real OTG Port, you could repurpose it as an Serial or even Virtual Ethernet port. And hell, Lady Ada striked again :)! So to sum it up shortly:
\nI build my MPICH as mentioned in 0) on an Minibian Jessie image (SDCard running on an RPi B). Then I installed her special kernel:<\/p>\n


\ncd ~
\nwget http:\/\/adafruit-download.s3.amazonaws.com\/gadgetmodulekernel_151224a.tgz -o gadgetkernel.tgz
\ntar -xvzf gadgetkernel.tgz<\/p>\n

sudo mv \/boot\/kernel.img \/boot\/kernelbackup.img
\nsudo mv tmp\/boot\/kernel.img \/boot<\/p>\n

sudo mv tmp\/boot\/overlays\/* \/boot\/overlays
\nsudo mv tmp\/boot\/*dtb \/boot
\nsudo cp -R tmp\/boot\/modules\/lib\/* \/lib<\/p>\n

# Load virtual ethernet module on boot
\necho 'g_ether' | sudo tee --append \/etc\/modules<\/p>\n

# Add settings to network interfaces
\necho '
\nallow-hotplug usb0
\niface usb0 inet static
\naddress 192.168.7.2
\nnetmask 255.255.255.0
\nnetwork 192.168.7.0
\nbroadcast 192.168.7.255
\ngateway 192.168.7.1' | sudo tee --append \/etc\/network\/interfaces<\/p>\n

# Add DNS Server
\necho 'nameserver 8.8.8.8' | sudo tee --append \/etc\/resolv.conf<\/p>\n

# Some additional tweaks:
\nAdd<\/p>\n

# Turn HDMI Off
\n\/usr\/bin\/tvservice -o
\n# Turn HDMI Back On
\n#\/usr\/bin\/tvservice -p<\/p>\n

# Turn ACT LED Off on Pi Zero
\necho none | sudo tee \/sys\/class\/leds\/led0\/trigger
\necho 1 | sudo tee \/sys\/class\/leds\/led0\/brightness
\n<\/code><\/p>\n

to your \/etc\/rc.local before exit 0 to turn off the HDMI Interface on boot,
\nas well as the LED of the Pi Zero to use less energy. Found on:
\nhttp:\/\/www.midwesternmac.com\/blogs\/jeff-geerling\/raspberry-pi-zero-conserve-energy<\/a> and http:\/\/www.midwesternmac.com\/blogs\/jeff-geerling\/controlling-pwr-act-leds-raspberry-pi<\/a><\/p>\n

This was enough to create an Pi Zero Slave Image.
\nShutdown the RPi now with
\n
\nsudo shutdown -h now
\n<\/code>
\nremove the Power and insert the SDcard into your Pi Zero.<\/p>\n

On the Master Machine, I did following changes:
\n
\n# Add settings to network interfaces
\necho '
\nallow-hotplug usb0
\niface usb0 inet static
\naddress 192.168.7.1
\nnetmask 255.255.255.0
\nnetwork 192.168.7.0
\nbroadcast 192.168.7.255' | sudo tee --append \/etc\/network\/interfaces<\/p>\n

# Allow Ipv4 Forward
\necho 'net.ipv4.ip_forward=1' | sudo tee --append \/etc\/sysctl.conf<\/p>\n

# Install iptables
\nsudo apt-get install -y iptables<\/p>\n

# Define NATing rules
\nsudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
\nsudo iptables -A FORWARD -i eth0 -o usb0 -m state --state RELATED,ESTABLISHED -j ACCEPT
\nsudo iptables -A FORWARD -i usb0 -o eth0 -j ACCEPT<\/p>\n

# Save NAT rules \/ load iptables on interface up
\nsudo touch \/etc\/iptables_masq.rules
\nsudo chown pi:pi \/etc\/iptables_masq.rules
\nsudo iptables-save > \/etc\/iptables_masq.rules<\/p>\n

Add
\npre-up iptables-restore < \/etc\/iptables_masq.rules\nunder the eth0 section in the network interfaces\n\nsudo vi \/etc\/network\/interfaces\n\ni.e.\nauto eth0\niface eth0 inet dhcp\n pre-up iptables-restore < \/etc\/iptables_masq.rules\n\n( Info taken from: http:\/\/serverfault.com\/questions\/405628\/routing-traffic-on-ubuntu-to-give-raspberry-pi-internet-access<\/a> ) <\/p>\n

#After that, I shutdown the RPi via
\nsudo shutdown -h now
\n#removed power from it.
\n<\/code><\/p>\n

Then I attached the Pi Zero to the Hub of Pi B via an Micro USB Cable by using the Micro USB OTG Slot on the Pi Zero, connecting it to the Hub of the Pi Modell B. Next, I powered up the Pi B - and both booted.<\/p>\n

I pinged 192.168.7.2 - which was the IP of the Pi Zero - and it answered. Now I only had to use cat ~\/.ssh\/id_rsa.pub | ssh pi@192.168.7.2 \"mkdir .ssh;cat >> .ssh\/authorized_keys\" from Section 0 to get the SSH Key, created in Step 0 onto the Pi Zero and could use that to automatically login in into the Pi Zero.
\nWith the new IP of the RPi B and Pi Zero in the machinefile of mpich I could then use my both RPis with higher speed and nearly zero costs for cabeling and power :)!
\nThe clou: I don't need an additional powersupply for the Pi Zero - nor network adapters, RJ45 cabling, an switch - only one USB A to USB Micro cable per Pi Zero - and maybe an big, active USB Hub ;)!<\/p>\n

Now, I need to get more Pi Zeros - I plan on using an Modell B as Master with an big active USB Hub to support 4 Pi Zeros - or an Modell B+ with an REALLY BEFFY USB Supply to work them all the same RPi - but that would come down to trying this... And I got only one Pi Zero - so I need some more time (or some sponsors?) to get me more RPi Zeros to try and see, whether this approach does scale ;)!<\/p>\n

Best thing: This can also be used to try the awesome work of http:\/\/blog.hypriot.com<\/a>\/ to build an Docker Cluster from that - cool, ain't it?<\/p>\n

Merry Christmas :)!<\/p>\n

Yours,<\/p>\n

Nico<\/p>\n