Flashing OpenWrt onto the Cisco Meraki MR33 access point the hard way

Please be reasonable and do not send any modified or bricked devices back to Cisco for servicing or replacement. I cannot be held responsible for any potential damages, everything you do is at your own risk.

Recently I purchased a couple of used Cisco Meraki MR33 access points to upgrade my wireless network. Up until this point, most of my network equipment was still on 5GHz 802.11n 2x2 (300Mbps). While being sufficient for internet browsing, it really slows down local file transfers. One day I saw someone was selling decommissioned Cisco Meraki APs at pretty reasonable prices, and some research revealed this particular model is even supported in upstream OpenWrt. It turns out riptidewave93 has done some excellent work on bringing OpenWrt support to this device, and even produced detailed flashing instructions. Quick glances at the guide revealed that Cisco later locked down the device with a firmware update and removed the u-boot shell access entirely. This wasn’t so much of a big deal to me because someone posted an NAND dump in the thread discussing this locked u-boot, so even if the device I received have locked bootloaders, I could simply take off the NAND flash and reprogram it offline.

Some people might question the value of going through so much trouble just to get some functional wireless APs, and it’s a perfectly valid concern. Doing this would be a total waste of time for people whose time is worth significantly more than the price of a couple of wireless APs. However, I do have some free time at my disposal, and tinkering with embedded Linux systems is something I personally find fun and rewarding. Plus it’s rather difficult to find cheap consumer grade APs that have this kind of good build quality, support PoE, look nice and clean on the wall, and are available to me at heavily discounted prices. Moreover, this device has three Wi-Fi radios and a BLE radio. The extra Wi-Fi radio could be very useful for keeping old devices incompatible with WPA3 connected, without me having to keep WPA2 on the main radios indefinitely. The BLE radio is potentially useful for IoT, but I haven’t planned to do anything with it just yet.

Disassembling this device was fairly simple. Removing the four Torx T5 screws under the four rubber feet and some gentle prying was all I needed to remove the back cover.

The first MR33 I opened was a jackpot, because it still had the original unlocked U-Boot 2012.07-g97ab7f1 [local,local] (Oct 06 2016 - 13:07:25). The device must have been decommissioned for quite a while now, or it would have been automatically updated already. I won’t go through the flashing process just yet because it’s identical to the next one after bootloader replacement. Other than the problem I had with ubootwrite (which I will talk about below), it was smooth sailing. I backed up all the partitions and installed OpenWrt, and all was well.

Unfortunately, my luck ran out on the second device, and it had the locked U-Boot 2017.07-RELEASE-g78ed34f31579 (Sep 29 2017 - 07:43:44 -0700). The xyzzy escape sequence is no longer working, and the discussion thread on GitHub indicated the entire shell has been removed.

However, I was still wondering if the device would drop back into a shell if boot fails, because sometimes it’s possible to obtain a shell without even knowing the secret escape sequence by simply shorting out the data pins of the flash to make u-boot fail to load the kernel image. Generally speaking, shorting the IO pins with each other on the flash should not have any harmful effects, but it will successfully prevent the host from reading any valid data from the chip. I simply shorted pin 29 and pin 30 with some tweezers.

The idea here is to short the data pins immediately after u-boot starts and before it has a chance to read the kernel image from the NAND. This requires some precision timing and took me more than 10 tries to get it to work, because the NAND is pretty fast and the process of reading a tiny kernel image to memory is almost instant. Unfortunately this did not produce any meaningful results, and someone mentioned in the thread that there’s a possibility of blowing efuses by giving the board random input, so I gave up on this and decided to just pull the flash instead.

U-Boot 2017.07-RELEASE-g78ed34f31579 (Sep 29 2017 - 07:43:44 -0700)

DRAM:  242 MiB
machid : 0x8010001
Product: meraki_Stinkbug
NAND:  ONFI device found
ID = 1d80f101
Vendor = 1
Device = f1
128 MiB
Using default environment

In:    serial
Out:   serial
Err:   serial
machid: 8010001
ubi0: attaching mtd1
Nand Flash error. Status = 12336
NAND page read failed. page: 1800 status ffffffb6
Nand Flash error. Status = 12336
NAND page read failed. page: 1840 status ffffffb6
--------< huge flood of errors removed >--------
Nand Flash error. Status = 12336
NAND page read failed. page: fd80 status ffffffb6
Nand Flash error. Status = 12336
NAND page read failed. page: fdc0 status ffffffb6
ubi0: scanning is finished
UBI init error 22
Error, no UBI device/partition selected!
Wrong Image Format for bootm command
ERROR: can't get kernel image!
Error, no UBI device/partition selected!
Wrong Image Format for bootm command
ERROR: can't get kernel image!
resetting ...

In the GitHub discussion thread, people mentioned a couple of potential solutions, but they all seem to require either manually messing with OOB data and/or expensive specialty devices such as the 360-clip and usbjtag, both of which costs significantly more than what I got my MR33 for, so those are obviously not feasible for me. However, one particular reply stood out to me, because they seem to have reprogrammed the flash with an i.MX 6ULL based Linux board. This is very intriguing for me, as not only does Linux have userspace utilities to read/write raw NAND, u-boot can do that too, and I don’t even have to deal with OOB/ECC data manually anymore. Messing with mtd and devicetree setups won’t be necessary if I could get hold of some other embedded device that can boot from a different medium and read/write the NAND.

I was able to dig up another IPQ40xx based board that happens to boot from SPI NOR but loads its kernel and rootfs from a TSOP48 NAND. Cross compiling a usable copy of functional u-boot for this board was a rather painful process, but I’ll talk about that in a future post.

I am well aware that not everyone just has random dev boards lying around, and this solution might be even less practical/financially viable for most people than buying the 360-clip and a compatible programmer. This post is not meant to be a general tutorial, it’s more like an elaborate journal that I put out in hope it could be an interesting read for people who are interested in the process.

Pulling the flash from the MR33 was a relatively trivial process, but some preheating is still required. A good trick to desolder components on these type of boards with lots and lots of ground planes is to put a decent amount of bismuth solder on the solder joints you’re working with. Bismuth solder has a very low melting point at about 138C, so it will significantly lower the melting point of the lead-free solder onboard when they’re mixed, and the board will become much easier to work with. Not only that, bismuth solder also has a benefit of being lead-free as well. I was able to pull the chip off with my hot air station set to only about 300C. If this was just the lead-free solder, I’d probably have to crank the temperature up to 360C to reliably melt the solder, which is rather dangerous to the board.

The original NAND on the dev board was desoldered and replaced with two 24 pin ZIF connectors.

A friend of mine designed a TSOP48 socket adapter board last year, and it finally came in handy this time. This board simply makes a pin to pin connection from the ZIF connectors to the NAND socket, and has no other components onboard.

With the flash attached, I powered up the board and backed up the full flash by reading 32MB chunks and sending them to a local TFTP server. The 32MB limit has something to do with the memory map, and it’s out of my control. Loading 64MB of data starting from 0x84000000 (the default load address) would cause a prefetch abort and a reset because it’s writing to illegal addresses. I did not experiment with pushing the address forward to load more data into memory in one go, because I was uncertain if I would write into memory regions that are already in use.

(IPQ40xx) # nand read 0x84000000 0x0 0x2000000 && tftpput 0x84000000 0x2000000 32M-1.bin

NAND read: device 0 offset 0x0, size 0x2000000
 33554432 bytes read: OK
eth0 PHY0 Down Speed :10 Half duplex
eth0 PHY1 up Speed :1000 Full duplex
eth0 PHY2 Down Speed :10 Half duplex
eth0 PHY3 Down Speed :10 Half duplex
eth0 PHY4 Down Speed :10 Half duplex
Using eth0 device
TFTP to server 192.168.1.1; our IP address is 192.168.1.2
Filename '32M-1.bin'.
Save address: 0x84000000
Save size:    0x2000000
Saving: ################################################################
         #################################################################
         --------< excessive logs removed >--------
         ######################################################T ###########
         ###########
done
Bytes transferred = 33554432 (2000000 hex)

(IPQ40xx) # nand read 0x84000000 0x2000000 0x2000000 && tftpput 0x84000000 0x2000000 32M-2.bin

(IPQ40xx) # nand read 0x84000000 0x4000000 0x2000000 && tftpput 0x84000000 0x2000000 32M-3.bin

(IPQ40xx) # nand read 0x84000000 0x6000000 0x2000000 && tftpput 0x84000000 0x2000000 32M-4.bin

After backing up the flash contents, I read mtd8 on my first MR33 that is already running OpenWrt. mtd8 is the partition named u-boot here, or more traditionally named APPSBL by Qualcomm’s terminology.

[    0.917813] 12 fixed-partitions partitions found on MTD device qcom_nand.0
[    0.925132] Creating 12 MTD partitions on "qcom_nand.0":
[    0.931985] 0x000000000000-0x000000100000 : "sbl1"
[    0.938898] 0x000000100000-0x000000200000 : "mibib"
[    0.943425] 0x000000200000-0x000000300000 : "bootconfig"
[    0.948131] 0x000000300000-0x000000400000 : "qsee"
[    0.953745] 0x000000400000-0x000000500000 : "qsee_alt"
[    0.958302] 0x000000500000-0x000000580000 : "cdt"
[    0.963075] 0x000000580000-0x000000600000 : "cdt_alt"
[    0.967812] 0x000000600000-0x000000680000 : "ddrparams"
[    0.972890] 0x000000700000-0x000000900000 : "u-boot"
[    0.979043] 0x000000900000-0x000000b00000 : "u-boot-backup"
[    0.984316] 0x000000b00000-0x000000b80000 : "ART"
[    0.988376] 0x000000c00000-0x000007c00000 : "ubi"

I cut up the flash dump from both devices and compared all the partitions except the ubi one, it turns out only u-boot (APPSBL) was ever updated. Everything else stayed the same.

$ for i in {0..9}; do diff -s mtd$i ../OLD_UBOOT_NAND_BACKUP/nanddump/mtd$i; done
Files mtd0 and ../OLD_UBOOT_NAND_BACKUP/nanddump/mtd0 are identical
Files mtd1 and ../OLD_UBOOT_NAND_BACKUP/nanddump/mtd1 are identical
Files mtd2 and ../OLD_UBOOT_NAND_BACKUP/nanddump/mtd2 are identical
Files mtd3 and ../OLD_UBOOT_NAND_BACKUP/nanddump/mtd3 are identical
Files mtd4 and ../OLD_UBOOT_NAND_BACKUP/nanddump/mtd4 are identical
Files mtd5 and ../OLD_UBOOT_NAND_BACKUP/nanddump/mtd5 are identical
Files mtd6 and ../OLD_UBOOT_NAND_BACKUP/nanddump/mtd6 are identical
Files mtd7 and ../OLD_UBOOT_NAND_BACKUP/nanddump/mtd7 are identical
Binary files mtd8 and ../OLD_UBOOT_NAND_BACKUP/nanddump/mtd8 differ
Files mtd9 and ../OLD_UBOOT_NAND_BACKUP/nanddump/mtd9 are identical

u-boot-backup remaining the same is rather unexpected, so I did a hexdump of the file and quickly understood why: it’s completely unused.

$ hexdump -C mtd9
00000000  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00200000

The u-boot image read from the first device was placed on my TFTP server and then loaded into memory of the dev board. The region starting with 0x700000 of size 0x200000 was erased, then the old u-boot image fetched from TFTP was written to the NAND.

(IPQ40xx) # tftpboot mtd8
eth0 PHY0 Down Speed :10 Half duplex
eth0 PHY1 Down Speed :10 Half duplex
eth0 PHY2 Down Speed :10 Half duplex
eth0 PHY3 Down Speed :10 Half duplex
eth0 PHY4 up Speed :1000 Full duplex
Using eth0 device
TFTP from server 192.168.1.1; our IP address is 192.168.1.2
Filename 'mtd8'.
Load address: 0x84000000
Loading: #################################################################
         #################################################################
         #############
done
Bytes transferred = 2097152 (200000 hex)

(IPQ40xx) # nand erase 0x700000 0x200000

NAND erase: device 0 offset 0x700000, size 0x200000
Erasing at 0x8e0000 -- 100% complete.
OK

(IPQ40xx) # nand write 0x84000000 0x700000 0x200000

NAND write: device 0 offset 0x700000, size 0x200000
 2097152 bytes written: OK

After the NAND is programmed, it was soldered back onto the board, and the board was cleaned.

I powered up the board with the serial console attached and saw the familiar U-Boot 2012.07-g97ab7f1 [local,local] (Oct 06 2016 - 13:07:25) version string. Now the difficult part is finally over, all I needed to do was repeating the steps I went through with the first device.

This was actually the tricky part for me personally, mostly caused by some bad assumptions of mine. I totally didn’t realize riptidewave93’s version of ubootwrite was modified until having spent about half an hour messing with the original version of this script. In hindsight, I should have checked if the script contained the string xyzzy first. While trying to debug issues with the python script, I found not having any way to watch the serial console really inconvenient, so I came with this setup with three USB serial adapters. One is the main adapter that is used to communicate with the board, and the other two enable me to monitor what’s being sent and received on the Tx and Rx lines.

Do not connect the VCC pin to your serial adapter.

Leaving the antennas detached is more or less fine and probably won’t damage the RF circuitry in this case. Stock OpenWrt will never start transmitting anything with the radios by default, and I will certainly not enable the radios without connecting the antennas first. I recommend connecting all antennas back before powering up the board just to be safe.

The second problem I encountered was with python2.7. Since python2.7 is already EoL, people have started removing it from software repositories. The python2.7 interpreter itself is still widely available for compatibility reasons, but a lot of Linux distributions have already removed most, if not all python2.7 library packages and the python2.7 version of pip from their repositories. This means installing python-serial (the python2 flavor of pyserial) is now very difficult. I really didn’t want to use yet another Ubuntu 18.04 virtual machine for this, so I decided to get it from Nixpkgs.

Nix is really convenient for setting up this type of ad-hoc development environments, where I just need a couple of things in a disposable environment to test some other stuff with on occasions, without installing them in a system level package manager.

The latest 21.05 channel on nixpkgs does not have python2.7 packages anymore, so I had to add the older 20.09 channel to get python2.7 related packages.

nix-channel --add https://nixos.org/channels/nixos-20.09

With the channel added, I put these in my shell.nix, and ran nix-shell. Now I have the python2.7 version of pyserial, and can proceed with the flashing process.

{ pkgs ? import <nixos-20.09>
    { }
}:

pkgs.mkShell {
  nativeBuildInputs = [
    python27Packages.pyserial
  ];
}

Within the newly launched shell, I ran python2 ubootwrite.py --write=mr33-uboot.bin --verbose --serial=/dev/ttyUSB0 and started uploading the intermediary u-boot over serial. This is what the serial consoles look like:

After a couple of minutes of waiting, the intermediate u-boot is started, and I could finally proceed to the next step.

Progress 100%                            
Waiting for prompt...
Prompt '
STINKBUG # ' not received. Instead received '
## Booting '
COM: kernel from Legacy Image at 82000000 ...
   Image Name:   MR33-UBOOT-1
   Image Type:   ARM Linux Kernel Image (gzip compressed)
   Data Size:    171744 Bytes = 167.7 KiB
   Load Address: 86000000
   Entry Point:  86000000
   Verifying Checksum ... O
COM: K
   Uncompressing Kernel Image ... 
COM: OK
Using machid 0x8010001 from environment

Starting kernel ...

The last problem I had was with TFTP. There is a pretty comprehensive rant on this subject which is worth a read in my opinion. However, it’s not like I’ve got any other options, so TFTP it is. Hosting a temporary TFTP file server on Linux is surprisingly annoying, because some of the solutions I tried either require some tedious configuration process, or just flat out did not work. On Windows, there is the excellent tftpd64 project, but unfortunately there doesn’t seem to be anything similar to it on Linux. The solution that finally worked for me was py3tftp, as I was able to put up a functional TFTP server serving the current working directory by simply running sudo py3tftp -v -p 69.

Actively uploading the OpenWrt ramdisk image to the intermediary u-boot was much less painful, echo -e "binary\nput openwrt-19.07.6-ipq40xx-generic-meraki_mr33-initramfs-fit-uImage.itb" | tftp 192.168.1.1 seems to work just fine. I didn’t use the supplied image but opted to use an official stable build instead. Additionally, I was unaware of the existence of OpenWrt 19.07.7 back when I was flashing the device, or I would have used that instead.

After more waiting, I finally got into OpenWrt.

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.14.215 (builder@buildhost) (gcc version 7.5.0 (OpenWrt GCC 7.5.0 r11278-8055e38794)) #0 SMP Tue Jan 19 13:10:02 2021

Installing OpenWrt to the NAND is a rather trivial process. Since the entire NAND was imaged in the previous chapter, basically all that is required is to delete all UBI volumes except ART (radio calibration data), re-create the fail-safe partition with an OpenWrt ramdisk boot image, and performing a normal sysupgrade. The rest of the volumes will be automatically recreated by sysupgrade and all the free space I have freed by deleting old useless volumes from the Meraki firmware will be available to OpenWrt.

# Cleaning up old UBI volumes used by the factory firmware

root@OpenWrt:/# ubinfo -a | grep -e "Volume ID" -e "Size" -e "Name"
Volume ID:   0 (on ubi0)
Size:        137 LEBs (17395712 bytes, 16.5 MiB)
Name:        diagnostic1
Volume ID:   1 (on ubi0)
Size:        133 LEBs (16887808 bytes, 16.1 MiB)
Name:        storage
Volume ID:   4 (on ubi0)
Size:        80 LEBs (10158080 bytes, 9.6 MiB)
Name:        part.safe
Volume ID:   5 (on ubi0)
Size:        67 LEBs (8507392 bytes, 8.1 MiB)
Name:        rootfs-25-xxxxxxxxxxxxxxxxxxxxxxxxxxx
Volume ID:   6 (on ubi0)
Size:        5 LEBs (634880 bytes, 620.0 KiB)
Name:        ART
Volume ID:   7 (on ubi0)
Size:        19 LEBs (2412544 bytes, 2.3 MiB)
Name:        part.old
root@OpenWrt:/# ubirmvol /dev/ubi0 -n 0
root@OpenWrt:/# ubirmvol /dev/ubi0 -n 1
root@OpenWrt:/# ubirmvol /dev/ubi0 -n 4
root@OpenWrt:/# ubirmvol /dev/ubi0 -n 5
root@OpenWrt:/# ubirmvol /dev/ubi0 -n 7
root@OpenWrt:/# ubinfo -a | grep -e "Volume ID" -e "Size" -e "Name"
Volume ID:   6 (on ubi0)
Size:        5 LEBs (634880 bytes, 620.0 KiB)
Name:        ART

# The following files were transferred to /tmp over scp
# openwrt-19.07.6-ipq40xx-generic-meraki_mr33-initramfs-fit-uImage.itb
# openwrt-19.07.6-ipq40xx-generic-meraki_mr33-squashfs-sysupgrade.bin

# Recreating the failsafe UBI volume and writing the OpenWrt ramdisk boot image to it

root@OpenWrt:/# file="/tmp/openwrt-19.07.6-ipq40xx-generic-meraki_mr33-initramfs-fit-uImage.itb"
root@OpenWrt:/# size=$(cat "$file" | wc -c)
root@OpenWrt:/# ubimkvol /dev/ubi0 --size=$size --type=static --name=part.old
Volume ID 0, size 53 LEBs (6729728 bytes, 6.4 MiB), LEB size 126976 bytes (124.0 KiB), static, name "part.old", alignment 1
root@OpenWrt:/# ubiupdatevol /dev/ubi0_0 "$file"
root@OpenWrt:/# ubinfo -a | grep -e "Volume ID" -e "Size" -e "Name"
Volume ID:   0 (on ubi0)
Size:        53 LEBs (6729728 bytes, 6.4 MiB)
Name:        part.old
Volume ID:   6 (on ubi0)
Size:        5 LEBs (634880 bytes, 620.0 KiB)
Name:        ART

# Performing a normal sysupgrade

root@OpenWrt:/# sysupgrade -v /tmp/openwrt-19.07.6-ipq40xx-generic-meraki_mr33-squashfs-sysupgrade.bin
Cannot save config while running from ramdisk.
Commencing upgrade. Closing all shell sessions.
Watchdog handover: fd=3
- watchdog -
killall: telnetd: no process killed
Sending TERM to remaining processes ... netifd odhcpd uhttpd ntpd ubusd dnsmasq urngd logd rpcd 
Sending KILL to remaining processes ... 
Performing system upgrade...
Volume ID 1, size 23 LEBs (2920448 bytes, 2.7 MiB), LEB size 126976 bytes (124.0 KiB), dynamic, name "part.safe", alignment 1
Volume ID 2, size 25 LEBs (3174400 bytes, 3.0 MiB), LEB size 126976 bytes (124.0 KiB), dynamic, name "rootfs", alignment 1
Set volume size to 97263616
Volume ID 3, size 766 LEBs (97263616 bytes, 92.7 MiB), LEB size 126976 bytes (124.0 KiB), dynamic, name "rootfs_data", alignment 1
sysupgrade successful

Now OpenWrt has been successfully installed to the NAND, and all the space I freed up is available to the system. There are some size discrepancies with the above output, because this is taken from a device running a more recent image built by myself, with some more things added on top.

root@OpenWrt:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 3.8M      3.8M         0 100% /rom
tmpfs                   121.4M    224.0K    121.2M   0% /tmp
/dev/ubi0_3              82.5M     60.0K     78.2M   0% /overlay
overlayfs:/overlay       82.5M     60.0K     78.2M   0% /
tmpfs                   512.0K         0    512.0K   0% /dev

With the software side properly taken care of, now I need to find a way to wall-mount these. Unfortunately, only one of the MR33s came with the factory mounting bracket. The essential part of this bracket is pretty much just the highlighted section. Additionally, the hooks on the spring-loaded section can be omitted as well, since a security screw will be used.

So I took some rough measurements with a caliper and traced a rough shape in OpenSCAD:

difference() {
    polygon([[0,0],[0,137],[4.95,137],[4.95,140],[14.65,140],[14.65,137],[19.6,137],[19.6,0]], paths=[[0,1,2,3,4,5,6,7,8]]);
}

This PNG illustration below is not perfectly to scale, it’s for dimension reference only. If you wish to send this to someone to cut it for you, please render the above source code in OpenSCAD and do a proper export from there.

The file exported from OpenSCAD was then sent to a metal laser cutting company. This is cut from 1.5mm aluminum sheets, which is just barely bendable by hand with pliers, and somewhat of an overkill for mounting such a lightweight AP. The bend job I did with pliers was pretty terrible, but that’s all I could do without a proper bench vice. In any case, this won’t be visible once the AP is mounted anyway, so it’s not a big deal. The screw holes were not included in the OpenSCAD source, because hand bending can cause a lot of variations, and it’s very unlikely that the wall-mount screw holes will be centered, and it’s even more unlikely that the security screw hole will be properly aligned at all. It’s best to bend it, fit it onto the AP, tighten the security screw a little to leave a mark on the bracket, and then drill a hole at that mark.

I placed a rubber foot near the top side of the bracket to keep the AP from tilting towards the wall.

About the security screw, it’s DIN7991, 2.5mm in diameter, and 12mm in length.

Flashing this device was unnecessarily complicated, but I learned quite a bit from this experience. Plus seeing all these decommissioned devices brought back to life with open-source firmware is a rather rewarding process. While the practice of aggressively locking down devices is really annoying from a tinkerer’s perspective, since this device is still officially supported by Cisco until 2026, it sadly does not make any financial sense for them to allow people to unlock the hardware and avoid the subscription service entirely.

Unfortunately, Cisco does not send out firmware updates to unlock devices after they are EoL, so without access to soldering tools and special NAND programmers, they will all end up as e-waste for most people. Perhaps it won’t matter much after 2026 since they will be long obsolete, but honestly I was still using 802.11n equipment up until now, so I can totally see myself using 802.11ac equipment few years down the line. Thanks to OpenWrt, most of the new CVEs related to Wi-Fi on the AP side can be patched or at least mitigated without having to change equipment.