Category: Computer

Computer Stuff

Cisco 45xx Supervisor 6LE Defect

As I tend to work with some Cisco 4506 and similiar switches - equipped with WSX45SUP6LE / WS-X45-SUP6L-E Supervisor Engines, I had the unfortunate delight to seem those rebooting at random.

Cisco did put out an entry in their bug tool ("Sup6LE reloads silently (Reset State: 00000201) / watchdog CISR0: 0x80") with the Bug Tracker ID CSCtf85481 - however, of the multiple accounts I had, I only found it on one of those. Other accounts did refuse my access to this file. As we confronted our Cisco Tech Support prior to finding the bug ourself, there was "no known error" which represented itself in rebooting the switch at random - so they said.

How do you find out it your board is one of those faulty ones?
a) If the serial number is JAE1422xxxx or higher (e.g., JAE1445xxxx, JAE1729xxx, etc) they board should not be affected. If it is in the range provided - check b)
b) If the "Hardware Revision" is 1.3 or higher, it is not affected. [sh idprom supervisor]
c) If either a) or b) are not true. Then the board may be affected by this bug. Contact TAC.

Another sign of the error is the "Last reload reason: Unknown reason" output of show version

It seems like the first batch of Supervisor 6L-E Engines were faulty on a HW level, so only a replacement will help. The engines will start rebooting once a while after a certain amount of time - with the time between each rebooting getting shorter.

Nexus 4 / Nexus 7 CrossUpgrade from Cyanogenmod to Lineage OS

EDIT: Experimental Builds were deleted now, as initial stated by LineageOS Team in this post. CrossUpgrade is not possible anymore, best bet would be to wipe the device and start from scratch. So just skip the second step with EXP Image and WIPE EVERYTHING in TWRP and install the latest nightly. Please also stay alert on bugs which are included in LineageOS. This link will give you access to the unresolved section of the Nexus 4 / Mako device, while this link will give you access to the same for the Nexus 7 2013 Wifi / flo :). Also, you can find the latest changes for mako here and for flo here.

As CyanogenMod ended their services - and Lineage OS took over, we need to cross-upgrade to the new system.

0.) Basic idea:
To upgrade to Lineage OS, they setup an so called experimental image. This exp image exists soley for the reason of migrating old CM installations to LOS installations. An really ugly-nag-screen will remind you of that. So the upgrade path will be: CM -> LOS Exp -> LOS Nightly. Also, you need to upgrade to the correct version. I.e. LOS does supply Nexus 7 LOS 14.1 images, so you need to upgrade your Nexus 7 first to CM 14.1, then install the LOS Exp 14.1, then the LOS Nightly 14.1

1.) Nexus 4 (mako)
- Install latest TWRP Version: https://twrp.me/devices/lgnexus4.html
- Make sure your device is on CM 14.1, if not, upgrade via: https://download.cyanogenmod.org/?device=mako
- Download LOS 14.1 Exp and install via TWRP (clean caches!) and reboot: https://download.lineageos.org/mako
- Download LOS 14.1 Nightly and install via TWRP (clean caches!): https://download.lineageos.org/mako

2.) Nexus 7 2013 Wifi (flo)
- Install latest TWRP Version: https://twrp.me/devices/asusnexus72013wifi.html
- Make sure your device is on CM 14.1, if not, upgrade via: https://download.cyanogenmod.org/?device=flo
- Download LOS 14.1 Exp and install via TWRP (clean caches!) and reboot: https://download.lineageos.org/flo
- Download LOS 14.1 Nightly and install via TWRP (clean caches!): https://download.lineageos.org/flo

And thats it! 🙂

More Infos needed? Just check out this older blog entry - you can use 2) and 3) but please do not delete anything except caches on 3) - and you do not need to flash GAPPS - as we want to upgrade, not reinstall from scratch 🙂

Odroid U3 Kernel Upgrade + Docker

I wrote this back in January 2017. Since then I had not much time to work on the Odroid - however, user hexdump did just came up with a new repo, supporting the Odroid U3 with Kernel 5.4.x - you can find the overview over his awesome work here and the repo with complete releases (i.e. Ubuntu Bionic or Debian Buster i.e. odroid_u3-armv7l-debian.img.gz) here

I am using an trusty old Odroid U3 which I acquired years ago. With its SAMSUNG Exynos 4412 Cortex-A9 Quad Core 1,7 Ghz, 1MB L2 cache and 2 GB of RAM, this little puppy was an real beast - compared to the Raspberry Pi 1 at that time. However, Hardkernel did drop the support - again, which left the Users back with very old Kernel versions. However, thanks to some users and the fact that all needed support for the Exynos is now included in the current kernel - well, we can build our own. This write up is the distilled result of days of work and a lot of research - and the work of other people which I found on the net (which I will try to give proper credits at the right locations :)).

EDIT: I upgraded the Kernel Configuration GIST for my Kernel Config + Docker on 10.02.2017. Thanks to an E-Mail from Tobias Jakobi I found the pieces I missed about adding the Kernel Internal Fanservice into the Config. This works now, however - I still like my tweaked program a bit better, as it cools the system more aggressivly, while the kernel default one is a lot more silent, but runs in the 80's°C while mine will stay at 70° on max load.

It is important that these instructions, especially if it comes down to installing stuff - is written for the usage of eMMC memory, NOT THE SDCARD! Also, there be dragons and something could go wrong - so please, as usual, advance at your own pace and risk! 🙂

0.) Get an Serial Interface for 1.8V
Important. The UART is 1.8V LVTTL ONLY! If you connect 3.3V or 5V, you'll blow the U3! I used an regular 5V TTL USB Adapter as well as an Sparkfun BiDir Level Converter: https://www.sparkfun.com/products/12009 With that set to 1.8V from the UART of the U3, it worked flawlessly with the usual 115000 BAUD.

Pinout:
http://odroid.com/dokuwiki/doku.php?id=en:u3_hardware

_____UART____
|Pin 4 - GND|
|Pin 3 - RXD|
|Pin 2 - TXD|
|Pin 1 - VCC|
___________|
1.8V LVTTL

1.) Build U-Boot
A lot of stuff is taken from here, thanks a lot for your great work, SnakeBite!
We asume you're working as root, as all this stuff will need root rights :).

# update your packages
apt-get update
# needed for building u-boot
apt-get install device-tree-compiler
# get ODROID signed u-boot
wget http://odroid.in/guides/ubuntu-lfs/boot.tar.gz
tar xzf boot.tar.gz
# get patched u-boot & build for the U3
git clone https://github.com/tobiasjakobi/u-boot
cd u-boot
make odroid_config
make
#copy fresh u-boot to ODROID directory
cp u-boot-dtb.bin ../boot/u-boot.bin
cd ../boot
## install on SDCard - not what we want, just as an remark for me
#bash sd_fusing.sh /dev/mmcblk0

Copy the needed files (u-boot.bin, E4412_S.bl1.HardKernel.bin, bl2.signed.bin, E4412_S.tzsw.signed.bin) to your PC, reboot your Odroid U3 into fastboot via connecting the UART to the U3 and aborting the boot. After that, you can issue the fastboot command on the UART. The U3 will now wait for filetransfer over the Micro USB Port, which you'll need to connect to your PC. Also, for the sake of an easy upgrade, use an Linux PC (more infos here: http://odroid.com/dokuwiki/doku.php?id=en:u3_building_u-boot ).

# install needed programs
sudo apt-get update
sudo apt-get install android-tools-adb android-tools-fastboot
# and - being in the right folder, start the transfer
# u-boot.bin install
sudo fastboot flash bootloader u-boot.bin
# bl1.bin install
sudo fastboot flash fwbl1 bl1.HardKernel
# bl2.bin install
sudo fastboot flash bl2 bl2.HardKernel
# tzsw.bin install
sudo fastboot flash tzsw tzsw.HardKernel
# If installation is done, you can reboot your ODROID-U3 with fastboot.
sudo fastboot reboot

You should now have a more recent U-Boot install.

Old: U-Boot 2010.12-svn (May 12 2014 - 15:05:46) for Exynox4412
New: U-Boot 2016.11-rc3-g8a65327 (Jan 07 2017 - 23:00:56 +0100)

By the way, we needed to download this boot.tar.gz, because it contains the keys needed to sign our new U-Boot install. More Infos about U-Boot and Keys: https://github.com/dsd/u-boot/blob/master/doc/README.odroid

The Installation of a more recent U-Boot version was necessary to facilitate the boot of the to-be-build new Kernel zImage with bootz.

1b.) eMMC recovery in case something goes wrong:
http://forum.odroid.com/viewtopic.php?f=53&t=969
DL the tool [ exynos4412_emmc_recovery_from_sd_20140629.zip ]

  1. Prepare a microSD card and flash the attached image.
  2. Insert microSD into U2/U3, disconnect eMMC
  3. Turn on U2/U3 and wait for a few seconds and blue LED will blink.
  4. Plug your eMMC module into U2/U3
    4b - wait 10 seconds!
  5. Plug micro-USB cable into U2/U3 and connect other side to your PC USB host or ODROID's USB host port. (This is a trigger to start the recovery)
  6. After recovery process (only a few seconds), the blue LED will turn off automatically.
  7. Finish. Install OS on your eMMC with as usual.

2.) Building Next Kernel for Odroid U3 with eMMC
And now to start the real work:

apt-get update
apt-get install live-boot u-boot-tools
cd ~
git clone --depth 1 git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git linux_odroid
cd linux_odroid
# we could make an default config, but this is not needed, we take rglinuxtech config in the next step
# make exynos_defconfig
# Odroid Config Kernel 4.4 from http://rglinuxtech.com/?p=1656
curl -o .config http://pastebin.com/raw/NveRajaZ
# Or you can use my Config which enables Docker as well (Gist at the End of the page)
curl -o .config https://gist.githubusercontent.com/nmaas87/81818c1db9dc292a4c21125bd2602658/raw/7e4e14fa15d7c68b177f31b9e2348d62c52cf83c/u3_docker_config
make menuconfig
make prepare modules_prepare
make -j4 bzImage modules dtbs
make modules_install
cp arch/arm/boot/dts/exynos4412-odroidu3.dtb /media/boot/exynos4412-odroidu3_next.dtb
cp arch/arm/boot/zImage /media/boot/zImage_next
cp .config /boot/config-`cat include/config/kernel.release`
update-initramfs -c -k `cat include/config/kernel.release`
mkimage -A arm -O linux -T ramdisk -C none -a 0 -e 0 -n uInitrd -d /boot/initrd.img-`cat include/config/kernel.release` /boot/uInitrd-`cat include/config/kernel.release`
cp /boot/uInitrd-`cat include/config/kernel.release` /media/boot/
cd /media/boot/
vi boot.txt
# now we have to rework the boot.txt / config
# comment out the old values and set in the new ones
# please do NOT copy blindly, you need to adjust the zImage, uInitrd and eyxnos4412***.dtb file names according to your system!
setenv initrd_high "0xffffffff"
setenv fdt_high "0xffffffff"
#setenv bootcmd "fatload mmc 0:1 0x40008000 zImage; fatload mmc 0:1 0x42000000 uInitrd; bootm 0x40008000 0x42000000"
setenv bootcmd "fatload mmc 0:1 0x40008000 zImage_next; fatload mmc 0:1 0x42000000 uInitrd-4.10.0-rc2-next-20170106-v7; fatload mmc 0:1 0x44000000 exynos4412-odroidu3_next.dtb; bootz 0x40008000 0x42000000 0x44000000"
#setenv bootargs "console=tty1 console=ttySAC1,115200n8 root=/dev/mmcblk0p2 rootwait ro mem=2047M"
setenv bootargs "console=tty1 console=ttySAC1,115200n8 root=/dev/mmcblk1p2 rootwait ro mem=2047M"
boot

#After you have done that, write the commands to the boot.scr file
mkimage -C none -A arm -T script -d boot.txt boot.scr
# sync and reboot and it should work
sync
reboot now

With this in mind I really upgraded my system from kernel 3.8.13 from 2015 - to the most recent 4.10.0-rc2 next Kernel 🙂

Old: Linux odroid 3.8.13.30 #1 SMP PREEMPT Fri Sep 4 23:45:57 BRT 2015 armv7l armv7l armv7l GNU/Linux
New: Linux odroid 4.10.0-rc2-next-20170106-v7 #3 SMP PREEMPT Mon Jan 9 19:17:32 CET 2017 armv7l armv7l armv7l GNU/Linux

2b.) FAN does not work, warning!
The CPU Fan does somehow not work right out of the box, so we will now enable it manually. [EDIT, with the new Kernel Config it works out of the box, but you can still decide to use this software to have a more aggressiv cooling :)]

# Fan to full speed
echo 255 > /sys/devices/platform/pwm-fan/hwmon/hwmon0/pwm1
# Read out current temperature in °C
cat /sys/devices/virtual/thermal/thermal_zone0/temp

To get things working again, I forked and updated the odroidu2 fan tool. Install it via:

git clone --depth 1 https://github.com/nmaas87/odroidu2-fan-service.git
cd odroidu2-fan-service
make
# install it as upstart service, i.e. < Ubuntu 16.04
make usi
# install it as systemd, i.e. Ubuntu 16.04 / Xenial
make systemd
reboot

Useful Commands:

# Read max CPU Speed:
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq
# Get current CPU Speed:
cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq
# Torrture Test:
openssl speed -multi 4

2c.) Upgrade to Xenial
As I upgraded to Xenial with do-relase-upgrade, I had some problems:

Authentication Problem:
It was not possible to authenticate some packages. This may be a transient network problem. You may want to try again later. See below for a list of unauthenticated packages. create /etc/update-manager/release-upgrades.d/unauth.cfg with

[Distro]
AllowUnauthenticated=yes

After upgrade, remove this file.
from: http://askubuntu.com/questions/425355/error-authenticating-some-packages-while-upgrade

After that, apt-get clean did not work:
apt-get clean
W: Problem unlinking the file apt-fast - Clean (21: Is a directory)

Solution was:

rm -rf /var/cache/apt/archives/apt-fast

from: http://askubuntu.com/questions/765274/error-problem-unlinking-in-apt-get-clean

2d.) MAC address changes every reboot:
One solution, which did not work, was following:

rm /etc/smsc95xx_mac_addr

from: http://forum.odroid.com/viewtopic.php?f=7&t=1070

Which worked better, was to really set the MAC address to a static value:
add in /etc/network/interfaces

auto eth0
iface eth0 inet dhcp
hwaddress ether bb:aa:ee:cc:dd:ff

from: http://forum.odroid.com/viewtopic.php?f=111&t=8198

2e.) Control the CPU speeds via cpufrequtils:

apt-get install cpufrequtils
vi /etc/default/cpufrequtils

ENABLE="true"
GOVERNOR="ondemand"
MAX_SPEED=1704000
MIN_SPEED=200000

However, I chose "performance" as GOVERNOR and a MIN_SPEED=800000

from: http://forum.odroid.com/viewtopic.php?f=65&t=2795

2f.) Install Docker
If you chose my .config with Docker enabled, you can install Docker with a fast
curl -sSL https://get.docker.com/ | sh
Thanks a lot to the Guys over at Hypriot, I took their RPi Kernel Configs as an example and merged those with the U3 Configs to get to this results. And yes, AUFS is still missing but... it is ok 😉

Additional stuff:
- Gist of my Kernel Config: https://gist.github.com/nmaas87/81818c1db9dc292a4c21125bd2602658

Following sites helped:
- https://blogs.s-osg.org/install-ubuntu-run-mainline-kernel-odroid-xu4/
- http://rtisat.blogspot.de/search/label/odroid-u3
- https://github.com/umiddelb/armhf/wiki/How-To-compile-a-custom-Linux-kernel-for-your-ARM-device
- http://rglinuxtech.com/?p=1622
- http://rglinuxtech.com/?p=1656
- http://forum.odroid.com/viewtopic.php?f=81&t=9342

[Docker] Upgrade Sonarqube from 5.6 to 6.2

I just updated Sonarqube from 5.6 (LTS) to 6.2. Before that, I upgraded all my plugins in Sonarqube itself and made an backup of my installation and database. Then, I replaced image: sonarqube:lts-alpine in my docker-compose.yaml with image: sonarqube:6.2-alpine. I did an docker-compose up and it started, however, I had some errors so it just kept on restarting. Following this advice I then proceded to delete $SONAR_HOME/data/es/. I then restarted Sonarqube, which worked. I then pointed my Webbrowser to the URL of my Sonarqube Instance and added an /setup and allowed the Database Upgrade. After that, I had a working new Sonarqube instance :).

Cisco RAM Problem (Phone/Linecard)

As a matter of fact, I've been working for more than 8 years with Cisco equipment and continue to do so. I really like Ciscos products, especially in the router / switch sector and had the pleasure to work with products in the range of Switching, Routing, Communications / Phones, Wifi, Datacenter Connectivity and Security. However, I had 3 unpleasent events with Ciscos products and I want to take the time to talk about two of those, as they occured because of the same reason.

If you don't know about Ciscos RAM problem, I want to give you a quick heads up: Fact is that Cisco installed defective SDRAM in almost all their products ranging from 2005-2012. The products with this defective RAM would work as normally, however, after being in use for more than 2 years AND an reboot, the products would fail - and stay that way. Cisco got to know about that problem in 2010, as they state themself, however, they informed users in 2012 for the first time. You can find out more about the topic on http://www.cisco.com/go/memory - this website was in 2014... As you can see, quite a lot of different products, including Routers like the 18xx/28xx series, Phones like the 79xx, the ASA55xx firewalls, Firewall Service Modules and more.

1.) Phones
As we had switched over to Cisco Phones a long time ago, we had multiple thousands of Cisco 79xx phones standing around and starting to die in 2014. We just got more an more messages from different customers that the phones just "went blank" and did not come up again. Only the speaker button was lit and thats it. As more and more phones died and we already opened up our own little graveyard, we went to Cisco with our problem - however, we never received an answer - until I figured out the problem myself: By disassembling some 7945, 7965 and 7975 - inspecting them and working around them with an self-made Serial Cable to the phones. It seemed like they would not start to unpack their image... As I figured the CPU should be fine an flash too, I came up with the theory that the SD-RAM was broken and found Ciscos website. However, I still insisted on proving my theory in the only way possible: Resurrecting one of our 7975 corpses from the graveyard.

I found the really good teardown on globalspec.com which stated that the SDRAM in this phone was a Samsung K4H561638H-UCB3 [SDRAM - DDR, 256Mb (16M x 16), 166 MHz, 2.5V, TSSOP 66]. After that I just removed the Motherboard from the Phone, removed the RAM with help from a friend (he got some really nice SMD reballing workstation :)) - and soldered in the new RAM. Without reflashing any Firmware or reset, it just worked after putting it back together! This proved my point.

(Picture was taken from http://electronics360.globalspec.com/article/3227/cisco-7975g-ip-phone-teardown)

2.) Linecards
Just some months ago, we had another accident with a linecard: One of our core switches rebooted due to power failure and after that, our 10 Gig Linecard, which connected one of our two main storage systems to the core, failed.

Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
  1    4  CEF720 4 port 10-Gigabit Ethernet      WS-X6704-10GE      xxxxxxxxxxx
  5    2  Supervisor Engine 720 (Active)         WS-SUP720-3B       xxxxxxxxxxx

Mod MAC addresses                       Hw    Fw           Sw           Status
--- ---------------------------------- ------ ------------ ------------ -------
  1  xxxxxxxxxxxxxx to xxxxxxxxxxxxxx   3.2   Unknown      Unknown      Other
  5  xxxxxxxxxxxxxx to xxxxxxxxxxxxxx   4.7   8.5(4)       12.2(33)SXH8 Ok

Mod  Sub-Module                  Model              Serial       Hw     Status 
---- --------------------------- ------------------ ----------- ------- -------
  1  Centralized Forwarding Card WS-F6700-CFC       xxxxxxxxxxx  4.1    Other
  5  Policy Feature Card 3       WS-F6K-PFC3B       xxxxxxxxxxx  2.7    Ok
  5  MSFC3 Daughterboard         WS-SUP720          xxxxxxxxxxx  2.12   Ok

Mod  Online Diag Status 
---- -------------------
  1  Unknown
  5  Pass
Router# show power
system power redundancy mode = redundant
system power redundancy operationally = non-redundant
system power total =     2771.16 Watts (65.98 Amps @ 42V)
system power used =       859.74 Watts (20.47 Amps @ 42V)
system power available = 1911.42 Watts (45.51 Amps @ 42V)
                        Power-Capacity PS-Fan Output Oper
PS   Type               Watts   A @42V Status Status State
---- ------------------ ------- ------ ------ ------ -----
1    WS-CAC-3000W       2771.16 65.98  OK     OK     on 
2    WS-CAC-3000W       2771.16 65.98  -      -      off
                        Pwr-Requested  Pwr-Allocated  Admin Oper
Slot Card-Type          Watts   A @42V Watts   A @42V State State
---- ------------------ ------- ------ ------- ------ ----- -----
1    WS-X6704-10GE       295.26  7.03   295.26  7.03  on    on
5    WS-SUP720-3B        282.24  6.72   282.24  6.72  on    on
6    (Redundant Sup)       -     -      282.24  6.72  -     -
Router#show platform hardware pfc mode
PFC operating mode : PFC3B

However, after replacing the Memory with new one, everything worked out - the Linecard was usable again!
I found information about the problem on Cisco again - after I resolved the problem: http://www.cisco.com/c/en/us/support/docs/field-notices/637/fn63743.html

The diagnostic test could be started with diagnostic start system test all

So, these are two problems I personally came across with Cisco Systems which failed, due to faulty memory and I decided to describe here - maybe some people stumble across these keywords and find the solution for their failing devices.

[Docker] Autobuild Docker Images from Github Repos you don't own

With Docker Hub it is dead simple to just link your Github Account to Docker, choose an Github Repo which contains your Dockerfiles and Apps - and build them as soon as you push new code. But as soon as you're not the owner of the repo (i.e. you're a fan and want to provide Dockerbuilds for Docker Hub) - things get messy. But actually, it works quite easy: Just apply some IFTTT - and you're set ;).

Ok, a little bit more detail: If this then that is a nice Website which lets you create some small but powerful Applets. I.e.: If the weather is cloudy, sent me an email. Things like that. Just sign up on https://ifttt.com, its free! 🙂

After that, just go over to https://hub.docker.com. Link your Github Account with Docker Hub and create and Auto Build Repo on Github with the Dockerfile of the project you want to build. In my case, the program I'd like is developed on Github itself, so my Dockerfile contains some instructions to just check out that repo, build the program and.. done :)! After this is working, go to the "Build Settings" Dialog of your Docker Hub Repo and go to the "Build Triggers" area. Activate the Build Triggers and Copy Down the Trigger URL. Every time the URL is used, your Repo is going to be build a new :).

The only thing we now need, is to know when a new commit is made. And Github makes that very easy: Go to your wanted Github Repo (I will use pklaus awesome brother_ql as example: https://github.com/pklaus/brother_ql), click on commits in the branch you like to use (i.e. master in my case) - and you will be redirected to the page showing you all commits: https://github.com/pklaus/brother_ql/commits/master. Just add an .atom after that URL, and voilá: You got yourself an RSS Feed of that commit stream ;)!. ( https://github.com/pklaus/brother_ql/commits/master.atom )

Copy that link as well, we're going to combine both links via the "magic" of IFTTT ;): Click on "my Applets", "New Applet" and on the "+ this" part. Select the Feed Service, and "New Feed item". Paste the collected Atom/RSS Feed URL in the Feed URL field and click "Create Trigger" (https://github.com/pklaus/brother_ql/commits/master.atom).

After that, click on the "+ that" part and search for "Maker" - and choose it as action service - the action should be "Make a web request". Now enter the Build trigger URL from Docker Hub you noted down, choose the "POST" Method, "application/json" as Content Type and enter "build": true as Body. After that, click save.

Activate your newly made Applet - and you're done: As soon as someone pushes a new commit, IFTTT will see that and start an rebuild of your Docker Hub Repo :).

(Yeah, that is a really hacked-together part, but it works ;))

Configure Git Line Endings

I tend to work crossplatform on different systems, mostly switching between different kinds of Ubuntu/Debian and Windows machines - always checking out and commiting git files on those machines. And I always forget to get the Git Line Endings right, as soon as I add a new machine to that list... So to do that, just a quick one liner:

git config --global core.autocrlf input
# Configure Git on Linux to properly handle line endings