WD My Cloud Mirror Gen2 with Debian 11 and Linux Kernel 5.15 LTS

Intro

Since 2017 I have been using an Western Digital My Cloud Mirror Gen 2 which I bought at Amazons Black Friday (or similar) - because the included 2x 8 TB WD Red were even cheaper with the NAS than standalone. Using the NAS had been quite ok, especially the included Docker Engine and Plex Support were a nice to have, the included Backdoor in older Versions - not so much. Recently WD had their new "My Cloud OS 5" replace the old My Cloud OS 3 - and made things worse for a lot of people. As I don't want any more surprises - and more control over my hardware - I decided to finally go down the road and get Debian 11 with an LTS (5.15) Kernel running on the hardware. This is how it went.

Warning

Warning, these are just my notes on how to convert a My Cloud OS 3 / My Cloud Mirror Gen 2 device to a "real" Debian system. You will need to take your device fully apart, solder wires and lose the warranty. Additionally you will lose all your data and even brick the hardware if something goes wrong, I am taking in no way responsibility, neither can I give support. You're on your own now.

Step 0: Get Serial Console Access

Without a serial console, you will not be able to do anything here. You will need to completely disassembly the NAS and will lose all warranty. The plain motherboard will look like this. On the most right side you will see the pins for the UART interface you will need to solder to.

When you're done with that, connect your 3v3 TTL UART USB device like this:

... and connect to it via 115200 BAUD with Putty, TeraTerm Pro or any other software (Do not connect the 3v3 pin :)). It would be wise starting without the hard drives installed.

Step 1: Flashing U-Boot

The current U-Boot on the NAS is flawed, you need to replace it. I will be CyberPK here which did an awesome job explaining everything:

We have to prepare an usb drive formatted in Fat32, and extract the uboot at link into it and connect to usb port#2.

Connect the device to the serial adapter, poweron the device and start pressing '1' (one) during the boot until you can see the 'Marvell>>' Command Prompt
press ctrl+c
then

We will start here to change stuff and break stuff. But if I could give you one tip before you start: Please execute printenv once. Copy and paste all env variables and everything Uboots spits out. It could save your hardware one day. Thanks, Nico out!

usb start
bubt u-boot-a38x-GrandTeton_2014T3_PQ-nand.bin nand usb
reset

This will reboot the device. Access again the Command prompt and add the following envs, a modified version of the ones provided by bodhi at this post:

setenv set_bootargs_stock 'setenv bootargs root=/dev/ram console=ttyS0,115200'

setenv bootcmd_stock 'echo Booting from stock ... ; run set_bootargs_stock; printenv bootargs; nand read.e 0xa00000 0x500000 0x500000;nand read.e 0xf00000 0xa00000 0x500000;bootm 0xa00000 0xf00000'

setenv bootdev 'usb'

setenv device '0:1'

setenv load_image_addr '0x02000020'

setenv load_initrd_addr '0x2900000'

setenv load_image 'echo loading Image ...; fatload $bootdev $device $load_image_addr /boot/uImage'

setenv load_initrd 'echo loading uInitrd ...; fatload $bootdev $device $load_initrd_addr /boot/uInitrd'

setenv usb_set_bootargs 'setenv bootargs "console=ttyS0,115200 root=LABEL=rootfs rootdelay=10 $mtdparts earlyprintk=serial init=/bin/systemd"'

setenv bootcmd_usb 'echo Booting from USB ...; usb start; run usb_set_bootargs; if run load_image; then if run load_initrd; then bootm $load_image_addr $load_initrd_addr; else bootm $load_image_addr; fi; fi; usb stop'

setenv bootcmd 'setenv fdt_skip_update yes; setenv usbActive 0; run bootcmd_usb; setenv usbActive 1; run bootcmd_usb; setenv fdt_skip_update no; run bootcmd_stock; reset'

saveenv

reset

(This code was also modified by me to use the fatload instead of the ext2load)

With this, our NAS is ready.

Step 2: Build a kernel and rootfs

  • On your current linux machine, get yourself a copy / git clone of Heisaths wdmc2-kernel Repo
  • Get all dependencies installed according to this repo, I installed it on a Debian 11 machine
  • Replace the file content of wdmc2-kernel/dts/armada-375-wdmc-gen2.dts with the content of the real and improved dts for the WDMCMG2 (original from this link, copy available here) - but keep the file name still armada-375-wdmc-gen2.dts
  • Replace the file content of wdmc2-kernel/config/linux-5.15.y.config with the file from here (please know this config ain't perfect, but it will get you running. You can always file a PR and help me out ;))
  • Start the build process in wdmc2-kernel with ./build.sh
  • Mark: Linux Kernel, Clean Kernel sources, Debian Rootfs, Enable ZRAM on rootfs
  • Kernel -> Kernel 5.15 LTS
  • Build initramfs -> Yes
  • Debian -> Bullseye
  • Fstab -> usb
  • Rootpw -> whateverYouWant
  • Hostname -> whateverYouWant
  • Locales -> no changes, accept (or whatever you want)
  • Default locale for system -> en_US.UTF-8 (or whatever you want)
  • Tzdata -> Your region
  • Now your kernel and rootfs will be build

While this is on-going, get yourself a nice USB 2.0 or USB 3.0 stick prepared with

  • partition table: msdos
  • 1st partition: 192 MB, FAT32, label set to boot, boot flag enabled
  • 2nd partition: rest, ext4, label set to rootfs

When the kernel is done compiling and your usb stick is done, copy all the files (sda is the name of my usb stick

  • mkdir /mnt/boot /mnt/root
  • mount /dev/sda1 /mnt/boot
  • mount /dev/sda2 /mnt/root
  • mkdir /mnt/boot/boot
  • cp wdmc2-kernel/output/boot/uImage-5.15.* /mnt/boot/boot/uImage
  • cp wdmc2-kernel/output/boot/uRamdisk /mnt/boot/boot/uInitrd
  • tar -xvzf wdmc2-kernel/output/bullseye-rootfs.tar.gz --directory=/mnt/root/
  • rm -rf /mnt/root/etc/fstab
  • cp /mnt/root/etc/fstab.usb /mnt/root/etc/fstab
    // within /mnt/root/etc/fstab:
    // change all /dev/sdb to /dev/sdc if all two drive slots on the NAS are used <- this!
    // change all /dev/sdb to /dev/sda if no drive slots on the NAS are used
  • umount /mnt/boot /mnt/root

Step 3: First boot and getting things running

Insert the USB stick into the 2 slot of the NAS. Leave the drives still out and boot it up for the first time, watch it via terminal. Login at the end with root and your chosen password.

If it boots, you can shut it down again with shutdown -P now, unplug power, insert the drives and reboot.

First thing after the first boot with drives, your own initramfs / Ramdisk from your current setup:

  • cd /root/
  • ./build_initramfs.sh
  • cp initramfs/uRamdisk /boot/boot/uInitrd

Second, install MDADM for the RAID:

  • apt update
  • apt install mdadm
  • mkdir /mnt/HD
  • edit your /etc/fstab and add a mount point for your md/raid. I used the old drives with my old data on it like this (depending on the fact as which mdX it launches...)
/dev/md0        /mnt/HD         ext4    defaults,noatime,nodiratime,commit=600,errors=remount-ro        0       1
 

A lot of good knowledge about Ramdrives can be found here.

I would advise to do steps: 1. Folder2RAM, 2. Kernel Options, 4. Logrotate - option 3 did not work out for me.

To get the drive to sleep at some point, we need to reconfigure MDADM

dpkg-reconfigure mdadm
// monthly check ok 
// daily degration check ok
// monitoring disable

... and get hdparm working

apt install hdparm hd-idle

# hdparm config
, add in /etc/hdparm.conf 

/dev/sda {
#        apm = 127
#        acoustic_management = 127
        spindown_time = 120
#       spindown_time = 4
        write_cache = on
}

/dev/sdb {
#        apm = 127
#        acoustic_management = 127
        spindown_time = 120
#       spindown_time = 4
        write_cache = on
}

# Spindown Time means: 120 * 5 sec = 600 sec / 60 sec = 10 min
# apply it after saving the file with:
/usr/lib/pm-utils/power.d/95hdparm-apm resume

We can check the status of the drives with smartctl

smartctl -i -n standby /dev/sda
smartctl -i -n standby /dev/sdb

To get fan control working

apt install wget
wget -O mcm-fancontrol-master.tar.gz https://github.com/nmaas87/mcm-fancontrol/archive/refs/heads/master.tar.gz
tar -xvzf mcm-fancontrol-master.tar.gz
cd mcm-fancontrol-master/
cp fan-daemon.py /usr/sbin/
chmod +x /usr/sbin/fan-daemon.py
cp fan-daemon.service /etc/systemd/system/
systemctl enable fan-daemon
systemctl start fan-daemon

(You can change low temp and high temp in the /usr/sbin/fan-daemon.py to get the Fan to kick in later and also set DEBUG = True if you want to see some details in the systemctl status fan-daemon)

MDT Utils can be useful, just mentioning it here

apt install -y mtd-utils
cat /proc/cmdline
cat /proc/mtd

Samba ...

apt install samba --no-install-recommends
# change /etc/samba/smb.conf to your liking and setup your SMB

Plex ...

# Plex 
apt update
apt install apt-transport-https ca-certificates curl gnupg2
curl https://downloads.plex.tv/plex-keys/PlexSign.key | apt-key add -
echo deb https://downloads.plex.tv/repo/deb public main | tee /etc/apt/sources.list.d/plexmediaserver.list
apt update
apt install plexmediaserver
systemctl status plexmediaserver

Well, that's it.

Thanks a lot to all awesome contributors in the net:

Companion repo with files: https://github.com/nmaas87/WDMCMG2

[WSL2/Win10] virt-manager for kvm on Windows

kvm Virtualisiation is great, however, useable tools to create and manage said VMs are rare. The best tool for the job, virt-manager is only available for Linux machines. But what if you want to manage said VMs also via Windows 10? WSL2 to the rescue: Just install WSL2 as shown by the excellent Microsoft Guide, install i.e. a Debian/GNU Linux instance and then launch into it.

You should update the instance to the latest version first:

sudo apt update && sudo apt upgrade -y && sudo apt dist-upgrade -y

then you can install virt-manager

sudo apt install -y virt-manager ssh-askpass

the last thing you would need is to install an X server on your windows machine, i.e. Xming or MobaXterm (which contains Xming) and launch it. Then you need to setup the X forwarding in your WSL2 instance, by entering

export DISPLAY="grep nameserver /etc/resolv.conf | sed 's/nameserver //':0"
export DISPLAY="sed -n 's/nameserver //p' /etc/resolv.conf:0"
export DISPLAY=$(ip route|awk '/^default/{print $3}'):0.0

after that, you can launch virt-manager by entering

virt-manager

and configure it to connect to your KVM instance via SSH.

Ubuntu 20.04 Update bricked KVM Virt

I updated an older Ubuntu 18.04 LTS system to the latest LTS and had (among other things) Docker and KVM installed. KVM is actually quite nice if you "just need" a small VM (pfsense ;)). I actually prefer Proxmox and ESXi, but hey, the right tool for the right job.

After the upgrade to 20.04, kvm did not work anymore and I got a lot of lvm2 errors during apt update / apt upgrade sessions, so a short google later I found this. I was a bit nervous, but the fix did neither hurt my kvm nor my Docker instances

sudo apt purge lvm2 && sudo apt install lvm2

(The fix is deleting and reinstalling lvm2)

After reinstalling lvm2, I could successfully execute a virsh list and got my list of running KVM machines back:

 Id   Name      State
-------------------------
 1    pfsense   running

Bash for sending and receiving raw IP packets

I stumbled across this feature during my bachelor studies:

echo "Hello World" > /dev/tcp/127.0.0.1/5000
echo "Hello World" > /dev/udp/127.0.0.1/5000

You need to be root (obviously) and its supported in bash environment - but not on all systems. You can also cat on the ports and use dns adresses. Its neat to just get a byte out :).

And if you need something more sophisicated, be sure to use the good old netcat ("nc")

SonarQube 6.7 Community with Postgres 9.6 in Docker on Ubuntu

This is a very quick install for SonarQube on Ubuntu 18.04 LTS. I presume you got the latest Docker CE 18.09 and docker-compose 1.24 installed.

# create folders for sonarqube files and postgres
sudo mkdir -p /var/sonarqube/{conf,data,logs,extensions}
sudo chown -R 999:999 /var/sonarqube
sudo mkdir -p /var/sonarqube/postgres
# make folder for all Docker files in home
mkdir ~/sonarqube
cd sonarqube
# create docker-compose.yml with following content
version: '3.1'
services:
  db:
    image: postgres:9.6-alpine
    restart: unless-stopped
    volumes:
      - /var/sonarqube/postgres:/var/lib/postgresql/data
    environment:
     - POSTGRES_USER=sonar
     - POSTGRES_PASSWORD=sonar

  sonarqube:
    image: sonarqube:6.7-community
    ports:
      - 9000:9000
      - 9092:9092
    restart: unless-stopped
    volumes:
      - /var/sonarqube/conf:/opt/sonarqube/conf
      - /var/sonarqube/data:/opt/sonarqube/data
      - /var/sonarqube/logs:/opt/sonarqube/logs
      - /var/sonarqube/extensions:/opt/sonarqube/extensions
    environment:
      - SONARQUBE_HOME=/opt/sonarqube
      - SONARQUBE_JDBC_USERNAME=sonar
      - SONARQUBE_JDBC_PASSWORD=sonar
      - SONARQUBE_JDBC_URL=jdbc:postgresql://db/sonar
# launch 
docker-compose up -d

You can then access your SonarQube instance on http://<ServerIP>:9000 with the credential admin/admin.

 

Migrate SonarQube from MySQL to PostgreSQL

For checking the quality of my private programming code, I have been using the free edition of SonarQube for multiple years. It is actually a cool project with a massive flaw: They did allow to use MySQL as Database - but not MariaDB. This struggle kept me a bit at bay, as I parted with MySQL years ago - with this one exception.

Some days ago I then stumbled upon the anouncement of SonarQube that did not fix this long-standing issue, but will be parting completly from MySQL. So the only choice would be to migrate to Oracle, MS SQL or PostgreSQL.

They even provided a tool called mysql-migrator for this purpose. However, this did not work. It always kept on complaining that it could not detect the schema version, etc.

Long story short: If you want (or in my case must...) switch from MySQL to PostgreSQL, use pgloader, which is available as Ubuntu package - all infos here.

I just installed a fresh PostgreSQL 9.6 in Docker, bundeled it with the postgres 9.6 alpine, fired it up and then migrated it with the following command:

pgloader mysql://<mysqluser>:<mysqlpassword>@<mysqlserverip>:3306/sonar pgsql://<psqluser>:<psqlpassword>@1<psqlserverip>/sonar

Due to all things Docker, one had to play a bit around with the correct IPs, Ports and permissions to get the tool working, but once that was said and done, everything worked fine:

                    table name       read   imported     errors      total time
------------------------------  ---------  ---------  ---------  --------------
               fetch meta data        180        180          0          0.357s
                Create Schemas          0          0          0          0.001s
              Create SQL Types          0          0          0          0.004s
                 Create tables        106        106          0          2.842s
                Set Table OIDs         53         53          0          0.015s
------------------------------  ---------  ---------  ---------  --------------
            sonar.active_rules       1993       1993          0          0.116s
  sonar.active_rule_parameters        268        268          0          0.107s
             sonar.ce_activity          1          1          0          0.074s
      sonar.ce_scanner_context          0          0          0          0.040s
           sonar.ce_task_input          0          0          0          0.025s
     sonar.analysis_properties          0          0          0          0.157s
      sonar.duplications_index          0          0          0          0.020s
                  sonar.events        349        349          0          0.173s
                  sonar.groups          2          2          0          0.330s
             sonar.group_roles         12         12          0          0.433s
                sonar.ce_queue          0          0          0          0.033s
                  sonar.issues       7508       7508          0          1.546s
 sonar.ce_task_characteristics          0          0          0          0.029s
       sonar.default_qprofiles          9          9          0          0.162s
                sonar.es_queue          0          0          0          0.143s
            sonar.file_sources        500        500          0          1.733s
        sonar.loaded_templates         13         13          0          1.498s
                 sonar.metrics        246        246          0          1.564s
           sonar.organizations          1          1          0          1.744s
           sonar.org_qprofiles         26         26          0          1.722s
   sonar.perm_templates_groups          4          4          0          1.724s
            sonar.groups_users          3          3          0          1.283s
sonar.perm_tpl_characteristics          0          0          0          1.673s
                sonar.projects        542        542          0          1.850s
     sonar.internal_properties          2          2          0          1.384s
           sonar.issue_changes        501        501          0          1.522s
         sonar.manual_measures          0          0          0          1.306s
           sonar.notifications          0          0          0          1.297s
    sonar.organization_members          2          2          0          1.504s
           sonar.project_links          0          0          0          1.532s
    sonar.permission_templates          1          1          0          1.453s
       sonar.project_qprofiles          0          0          0          1.483s
    sonar.perm_templates_users          0          0          0          1.417s
        sonar.qprofile_changes       2001       2001          0          1.616s
                 sonar.plugins         13         13          0          1.366s
     sonar.qprofile_edit_users          0          0          0          1.545s
        sonar.project_branches          3          3          0          1.356s
 sonar.quality_gate_conditions          4          4          0          1.353s
          sonar.rules_metadata       1763       1763          0          1.311s
        sonar.project_measures      35940      35940          0          1.792s
          sonar.rules_profiles         26         26          0          1.503s
       sonar.schema_migrations        495        495          0          1.369s
                   sonar.users          3          3          0          1.497s
             sonar.user_tokens          5          5          0          1.489s
              sonar.properties          8          8          0          0.941s
    sonar.qprofile_edit_groups          0          0          0          0.924s
           sonar.quality_gates          1          1          0          0.834s
                   sonar.rules       1866       1866          0          1.069s
        sonar.rules_parameters        278        278          0          1.021s
       sonar.rule_repositories         21         21          0          0.999s
               sonar.snapshots        280        280          0          1.047s
              sonar.user_roles          0          0          0          0.960s
      sonar.webhook_deliveries          0          0          0          0.942s
------------------------------  ---------  ---------  ---------  --------------
       COPY Threads Completion          4          4          0          3.049s
                Create Indexes        127        127          0         22.308s
        Index Build Completion        127        127          0          1.460s
               Reset Sequences         33         33          0          0.051s
                  Primary Keys         51         51          0          0.051s
           Create Foreign Keys          0          0          0          0.000s
               Create Triggers          0          0          0          0.001s
              Install Comments          0          0          0          0.000s
------------------------------  ---------  ---------  ---------  --------------
             Total import time      54690      54690          0          8.412s

 

BeagleLogic

To get your Beaglebone Black (BBB) to work as an Logic Analyzer, you can use BeagleLogic:

1.) Download latest version of BeagleLogic
from: https://beaglelogic.readthedocs.io/en/latest/beaglelogic_system_image.html
Released on 2017-07-13, sha256sum = be67e3b8a21c054cd6dcae7c50e9e518492d5d1ddaa83619878afeffe59c99bd
https://goo.gl/RiXGBs

2.) Burn it onto sdcard with Etcher

3.) Put SDCard into BeagleBone Black

4.) Hold "Boot" Button, plug in BBB with Mini USB Cable to your PC

5.) After 10 seconds or so, release "Boot" Button (this will just boot from SDcard with this trick)

6.) Wait some time and login to 192.168.7.2 via SSH
Username: debian
password: temppwd

7.) Test if BeagleLogic works
ls -l /dev/beaglelogic
There should be entry /dev/beaglelogic as answer, if not check the BeagleLogic Website

8.) Change uEnv.txt
sudo vi /boot/uEnv.txt
#go to following lines:
##enable Generic eMMC Flasher:
##make sure, these tools are installed: dosfstools rsync
#cmdline=init=/opt/scripts/tools/eMMC/init-eMMC-flasher-v3.sh

remove # from #cmdline=init=/opt/scripts/tools/eMMC/init-eMMC-flasher-v3.sh

#save and exit

9.) Shutdown
sudo shutdown now

10.) Boot again, this time with flasher
Hold "Boot" Button, plug in BBB with Mini USB Cable to your PC, after 10 seconds, release the button.
During the flash process of the eMMC, the 4 leds will make a "Knight Rider" like lightshow.
This will take some minutes. The BBB will shutdown afterwards (all LEDs off).

11.) Remove SDCard, replug Mini USB and boot without holding any buttons

12.) Wait some time and login to 192.168.7.2 via SSH
Username: debian
password: temppwd

13.) Open your webbrowser and go to: http://192.168.7.2:4000/#

Next:
- Upgrade
- this enables you to also use the BeagleLogic as "external analyzer" and control it from i.e. a Window System running a special version of pulseview 🙂
- More infos: https://goo.gl/L9QUrt
Connect to Ethernet / Internet
cd /opt/BeagleLogic/
git config http.sslVerify false
git pull
sudo apt update
sudo ./install.sh --upgrade
sudo reboot now

Useful infos:
https://beagleboard.org/getting-started#update
http://beagleboard.org/latest-images
https://beaglelogic.readthedocs.io/en/latest/beaglelogic_system_image.html
https://beaglelogic.readthedocs.io/en/latest/install.html

Debian 9 Offline Installation Bug

I had to install a Debian 9.3 without having access to the internet, so I downloaded the full size offline install DVD from https://cdimage.debian.org/debian-cd/current/amd64/iso-dvd. As usual, I inserted the first DVD into the DVD Drive, started the install and everything worked fine - except for the moment where it didn't ;): I needed to switch DVDs for some software - and directly afterwards, Debian came to the "Install Grub" step - and asked politly to insert DVD 1 again - which I tried to do - but could not:

For some reason, the Debian GUI Installer had locked the DVD Drive and I could not remove the DVD - could not open it. I then used the Emergency Eject, inserted DVD 1 and closed the Drive again with very mild force.

However, the Installer did not recognize the new DVD and said, that I needed to insert a DVD. Well...

To cope with that problem, just CTRL+ALT+F1 switch to a different Terminal, enter mount /dev/cdrom /cdrom and switch back to the Installer Terminal with CTRL+ALT+F7 (I think it was ;)) - and hit "Continue" - and from that moment on, it worked again and the installation ended successful after the GRUB installation.

 

[Dell] T30 Intel AMT Blank Screen on Ubuntu Fix

The Dell T30 is an awesome little Homeserver, packing a punch with the Xeon E3-1225 V5 - and being affordable at about 399 €. It also comes with Intels Active Management Technology / AMT which is an extension of the horrible Intel Mangement Engine (which was all over the place months ago when some genius figured out how to stop that Man-in-the-Middle-always-on chip with some simple commands) - but quite useful - nonetheless. The good thing about this, is that it acts like an DRAC (Dell) / ILOM (Sun) / IPMI (Supermicro) card - so it is an KVM (Keyboard Video Mouse, not the virtualization thingy this time, sorry ;)) extension which allows you to control the server via network as if you were plugged in directly.

There is an awesome guide from Christian on goNeuland, written in German on howto setup that thing without the need to buy VNC Viewer Plus.

However, my Ubuntu instance came in as blank screen after successfully connecting to the system. In the end, that turned out to be that way, as Ubuntu decided to deactivate the graphics unit - due to no monitor being attached.

Different solutions were talked about herehere and here.

In my case, following helped:

1.) Open your grub, i.e. sudo vi /etc/default/grub file

2.) Add nomodeset to your GRUB_CMDLINE_LINUX_DEFAULT line, so that it would read i.e. GRUB_CMDLINE_LINUX_DEFAULT="reboot=force bootdegraded=true nomodeset" (your commands will vary!)

3.) Save and close the file

4.) Update grub via sudo update-grub

And after a quick reboot, everything worked out :)!