Reverse-Engineering on The Dark Programmer's Tome of Secrets

Jailbreaking an All in One ISP Router: Sagemcom F@ST 5655v2

Sun, 16 Feb 2025 00:00:00 +0000

The problem

With the rapid expansion of high-speed access, Fiber-to-the-Home is becoming the standard for providing connectivity to urban residences. While FttH provides great benefits when it comes to speed and reliability, it is usually accompanied by a renunciation of our right to handle our own infrastructure.

Most ISPs that install FttH provide All-in-One devices as the CPE, which does both the Fiber-to-Ethernet modulation and the typical router / switch / access point tasks we nowadays expect from our routers. This device usually has a very locked down set of features. Hardly ever you will see any VLAN or firewall configuration available to the end-user, for example.

This is a very comfortable setup for the non-technically inclined, but it becomes excessively burdensome for any task that requires a minimum of customization.

My case is not out of the ordinary in this regard. Around 2019, our fiber connection was serviced by MasMovil, which kindly also supplied us with a Sagemcom F@ST 5655v2, an AIO router with moderate specs, enough to service our 300MbE connection.

At the time, I started reading up on IPv6. I found it terribly interesting and was wondering why its adoption was so lacking, especially in Spain, which averaged around 3% at the time, trailing all big european countries except Ukraine.

I evidently also read about Hurricane Electric, an internet service provider that specializes in IPv6 connectivity. They are one of the biggest providers of IPv6 transit, being considered a tier 1 network in this protocol. They are also an IPv6 tunnel broker: they provide individuals that lack native IPv6 with IPv6 tunnels to be able to access the IPv6 internet.

Here is the catch: To be able to use these tunnels, you require a router that can be configured for such a task, which is to run a 6in4 tunnel. The router that MasMovil provided obviously did not allow for such a feature. Therefore, there was only one thing I could do: Figure out a way to replace it with my own.

This task would not prove easy. I quickly learned that the protocol that most ISPs use for last-mile connectivity is GPON, and these GPON connections require a password in order to decode the traffic and connect to them.

Alas, I did not have such password. Reading online I found that you can ask for such password when the technician first installs the equipment, but I did not know that at the time, so I did not think to ask then.

The only place that I could potentially obtain this password from is the router itself. I would need to access the router and extract this secret from inside.

First steps: search online

The obvious first reaction was to search online. There I found a few blog posts describing a way to open ssh access into several models of Sagemcom routers, such as the F@ST 5355 or 5364. I attempted to reproduce these attacks without luck. Searching a bit more it turned out that such attack was patched in the newest firmware version, which I had.

With such bad prospects, I chose to buy the same model off second-hand to replicate the attack and obtain more info on the internals of this machine to be able to craft a new attack.

Having booted this second-hand router, I successfully managed to apply the hack to gain ssh access. Unfortunately, there was no way of replicating this on the newer version.

Second strategy: UART

Just for fun I chose to pry open the plastic casing that protected the router PCB. I couldn’t make much sense of that was going on in there, but after looking around for a bit, I did see something familiar.

I managed to spot a set of 4 pads lined up and very clearly labeled. It had to be a UART connector. I already had little hope for this attack, but I had to try it anyway. I bought a cheap USB to UART converter and connected it to the board.

After rebooting the device, I was presented with the following log:

UART Log (Abridged) (Click to expand)

CFE version 1.0.38-116.174 for BCM96838 (32bit,SP,BE)
Build Date: Fri Jan 20 18:54:28 CET 2017 (g601671@rmm-1186759)
Copyright (C) 2000-2013 Broadcom Corporation.
Version cfe-ram: 7.31.17.1

U-Boot 2011.12
Version: 7.31.17.1-full (Jan 20 2017 - 18:53:37) 
Copyright (C) 2011 - 2013 Sagemcom All rights reserved
Board: Sagemcom fast
CPU: Broadcom BCM68380 (Chip1 Rev4)
DRAM:  256 MiB
NAND:  128 MiB
Using default environment

## Booting kernel from Legacy Image at 8c3df000 ...
   Image Name:   scOS SG3V10000295 (masmovil_v0.3
   Created:      2020-05-26   5:24:05 UTC
   Image Type:   MIPS Linux Kernel Image (gzip compressed)
   Data Size:    2424832 Bytes = 2.3 MiB
   Load Address: 80010000
   Entry Point:  803ba690
   Verifying Checksum ... OK
   Uncompressing Kernel Image ... OK

Starting kernel ...

Linux version 3.4.11-rt19 (g533718@rmm08492) (gcc version 4.6.2 (GCC) ) #1 SMP PREEMPT Tue May 26 07:23:57 CEST 2020

This log does not do much for the task at hand, but it does provide some interesting information.

We gather that it is running on U-Boot 2011.12, a GPL 2.0 licensed micro-bootloader. U-Boot by default would allow you to press Ctrl+C to abort the boot and drop you into a command prompt with useful commands, such as md (memory display) which allows for reading the contents of specific memory locations. I tried interacting with the terminal, but the only thing I got it to print was Got Ctrl when I pressed Ctrl+C, with no other reaction.

It seems Sagemcom might have modified their version of U-Boot to disable the command prompt, thus locking down any possibility of interacting through the UART. According to the GPL, they should provide the source to anyone that asks. After contacting them on three different email addresses, they refused to provide the GPL-licensed code, which means they’re in clear violation of the license. I afterwards found on the internet a squashfs image which included the supposed bootloader binary for a router in the same family. Just for fun, I did some decompiling with Ghidra, and what I found would not surprise you.

The following is the original u-boot source code:

// arch/mips/lib/board.c
void board_init_r(gd_t *id, ulong dest_addr)
{
   /* ... */
	puts("Net:   ");
	eth_initialize(gd->bd);

	/* main_loop() can return to retry autoboot, if so just run it again. */
	for (;;)
		main_loop();

	/* NOTREACHED - no way out of command loop except booting */
}

And the following I managed to decompile from the binary:

void board_init_r(gd_t *id, ulong dest_addr)
{
  /* ... */
  puts("Net:   ");
  eth_initialize(gd->bd);

  if (gd->unknown_3 != 0) {
    sagem::sb3_sagem_init();
                    /* WARNING: Subroutine does not return */
    hang();
  }

  do {
    main_loop();
  } while( true );
}

They modified the source very explicitly to avoid reaching the main_loop function, which handles the Ctrl+C command to drop into boot prompt.

It seemed like this avenue was dead.

Breakthrough: Impersonate the ACS server

I searched a bit more online and found a new article that managed to escape me at first. It is not specific to my router model, but at first glance it looked like it might do the trick.

The essence of this blog post’s attack is to impersonate the ACS (Application Configuration Service) server, which is responsible for configuring every aspect of the router remotely by the ISP. We impersonate it by acting as a proxy between the router and the real ACS server, thus being able to listen to their conversation. It looked promising at first, but I ran into a small roadblock. Whereas the post author’s model allows for this ACS server to be within the LAN, mine only allowed this connection through the fiber port.

I then had the idea of trying to use the second-hand router I got to impersonate the real router. Since the second-hand router was already cracked, I managed to change the interface in which the ACS connection would be established to an ethernet connection. In this way, I could set up a laptop to do the proxying between the second-hand router and the ACS server.

From this ordeal I got a rough understanding of what exactly was being communicated in this conversation: A huge amount of SOAP requests and responses. Some of them more than a Megabyte long. Most of their content was useless, but I got a rough understanding of how these communications were happening, which messages were sent in response to which requests, or what was the general content of these request and response bodies.

I felt ready to attempt the same on the real router. I spun up an EC2 instance with the same proxy script that I was using earlier, and set its ip address as the ACS url in the router. Before restarting the router, I did a simple smoke test to make sure it was working. I tried to send a request myself pretending to be the router, and the result discouraged me significantly. My EC2 instance could not connect to the ACS server. It seems they might be refusing any connections coming from outside their network.

Not falling for despair, I decided to act as the server myself. For every request that the router would send, I quickly crafted a fake response based on the communications I gathered earlier. In particular, I crafted the following response:

<soap:Envelope xmlns:cwmp="urn:dslforum-org:cwmp-1-0" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soap-enc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
   <soap:Header>
      <cwmp:ID soap:mustUnderstand="1">27b552c7</cwmp:ID>
   </soap:Header>
   <soap:Body>
      <cwmp:SetParameterValues>
         <ParameterList soap-enc:arrayType="cwmp:ParameterValueStruct[2]">
            <ParameterValueStruct>
               <Name>Device.Services.X_MM_RemoteAccess.SSHEnable</Name>
               <Value xsi:type="xsd:boolean">true</Value>
            </ParameterValueStruct>
            <ParameterValueStruct>
               <Name>Device.Services.X_MM_RemoteAccess.TELNETEnable</Name>
               <Value xsi:type="xsd:boolean">true</Value>
            </ParameterValueStruct>
         </ParameterList>
         <ParameterKey>n/a</ParameterKey>
      </cwmp:SetParameterValues>
   </soap:Body>
</soap:Envelope>

And it worked! This finally allowed me to force the server to open the ssh ports it has been guarding so strongly. The username and password were the same as the ones used for the web interface, which I already had. I was in! It was only one more step to obtain the GPON password:

$ cat /opt/filesystem1/data/optical_conf.txt
3455xxxxxxxxxxxxxxxx

Just for fun: Decompiling the firmware

We already captured the flag that we were looking for, but there was still a thing that bothered me. In order to let the user keep backups of the config, the web UI exposes a config backup page. Clicking the backup button downloads a file called device.cfg. From the start it was obvious they would not let anyone meddle with the router configuration that easily, so I was already expecting some hardships when opening this file. The reality was even worse. The file started with the magic byes AEAD. These aren’t really any standard format that I could find, but considering AEAD stands for Authenticated Encryption with Associated Data, this pointed in only one direction: The file was encrypted. I therefore took it as a challenge to decrypt it. Since I was already in full control of the router, I could easily download all the binaries that it was using to decompile them and try to reverse the encryption.

Searching for the magic bytes within the libraries led me straight to the source: libgsdf.so.1. Doing some decompiling of the library with Ghidra (which was made easier thanks to the debug symbols they graciously left in the binary) I found the function in which this string was being referenced. The function was called gsdfAeadEncrypt, and what lay inside was straight up the encryption key:

KEY[0] = '}';
KEY[1] = 0xa2;
KEY[2] = 'X';
KEY[3] = '\x13';
KEY[4] = 0xdd;
KEY[5] = 0x9d;
KEY[6] = 'z';
KEY[7] = '\x15';
KEY[8] = '>';
KEY[9] = '`';
KEY[10] = 0xa0;
KEY[11] = '(';
KEY[12] = 0xba;
KEY[13] = 0xdd;
KEY[14] = 0xb2;
KEY[15] = 0x88;

Which, converted to a more readable format, is 0x7da25813dd9d7a153e60a028baddb288. @Sagemcom: You can make me remove this from here if you give me the modified u-boot source code. Deal?

Addendum

After having done all of this back in 2020, someone else decided to independently discover how to extract the encryption key and posted their results on their blog, which cites other blog posts as their sources. Due to my procrastination, I only found out about this while writing this post 5 years after the fact. I still decided to publish this post because, hey, why not. I also did something that they did: Cross compile a small tool to run the decryption library in a more convenient way. Go read about it in their blog post! Although, in contrast as to how they did it, I actually did a cross-compilation from an x86 machine to the MIPS architecture of the router, which involved downloading and installing a cross-compilation toolchain.

Reverse engineering a mouse configurator

Mon, 28 Aug 2023 00:00:00 +0000

Preamble: The problem

Once upon a time I had a mouse. This little mouse had five buttons. The typical three, plus a forward and backward on the side. It was nothing fancy, but it got me hooked on mice with additional buttons. There’s something very satisfying about moving through your browser history with a single button that I could not relinquish. After this mouse broke seven years ago, in 2016, I bought a brand new Mars Gaming MM4 18-button mouse. It has a nice 12-button keypad on the side, which can be configured with a propietary tool for Windows.

I was in university at the time, and Microsoft gave us free Windows keys through the Inspire program. I was already a Linux fanboy, but I had to keep Windows installed to run some programs that were necessary for my university, such as Solidworks, so I put up with this mouse configurator for the time being.

After I was done with university, I chose to nuke the Windows partition, which left me with no way of modifying the button mapping or changing the DPI resolution profiles. I tried running the configurator through Wine, but it did not work. It would boot, but it would refuse to communicate with the mouse. I then did the only sensible choice and decided to reverse engineer the configurator tool, or more specifically, the USB communication protocol.

Reverse engineering the communication protocol

To be able to reverse engineer the communication protocol that the mouse uses, I would firstly need an easy way of intercepting and processing the communication between the computer and the mouse. This proved easy, since Wireshark, our beloved network packet sniffer, can be used to intercept USB communications¹. I then spent a couple of days using the configurator, tweaking a single parameter, capturing the USB output and analyzing it to find how that tweak was encoded in the wire.

The mouse configurator seemed to be sending and receiving a dozen or so packets to and from the mouse. For each of the tweaks I made, I saved the packets into a pcap file with the data before and after the tweak. To convert the pcap files into a more manageable file format I wrote a small script in python. This scirpt uses the pyshark library to open the files, extract the info I wanted and write it out as a plain csv, with the binary data displayed as hexadecimal. I then loaded these csv files into Libreoffice Calc for further processing.

Having the data before and after the tweak allowed me to very easily see the differences that resulted in the tweak being applied. It was a matter of a game “Find the differences”, only I have a computer to find them for me. Any information I uncovered from this analysis I wrote down into Ghidra, which has a convenient Data Type Manager which lets you define data types, their structure, size and add explanatory comments to them.

With this same procedure I managed to reverse engineer around 75% of all the data being sent, and almost all of the exposed features on the configurator tool, with the exception of complex macros². I was satisfied with the work I had put in, but all of this would be in vain if I didn’t use it to create my own configuration tool.

Implementing a new driver in ratbag

I don’t need to say this, but creating your own configuration tool from scratch is a very hard task. That’s why I chose to implement my findings into the ratbag project. Ratbag, plus its frontend Piper, are MIT/GPLv2 licensed tools for configuring input devices, chiefly mice. They are, as far as I know, the predilect tool for this kind of task in Linux.

Figuring my way around the repositories took a couple of days, but seeing how other drivers were implemented helped significantly. Implementing a new driver is a matter of implementing two main components: The probe component, which obtains the current running configuration of the mouse, and the commit component, which applies the configuration to the mouse. Implementing the commit component proved easy, since I simply had to replicate the behavior that I previously reverse engineered. I even made it more efficient by not commiting configuration that hasn’t been changed, which the propietary tool does not do.

I then went ahead implementing the probe component. There is only a slight problem here. The propietary mouse configurator hardly does any probing. It only probes the device id, likely to confirm that it’s a supported device. It does not probe the current configuration of the mouse, such as buttons, LEDs and DPI resolutions. The configuration that shows up when the tool is launched is the one that is saved to disk on the host machine.

It didn’t take long while messing around to adapt the protocol to add this new feature. I analyzed the packet that probes the device id and identified the section that allowed for reading values from the mouse. Luckily, the same technique can be applied to the rest of the configurable values so that they can be read.

In parallel, I was also doing some changes in the Piper project. I needed to create an svg that resembled the mouse that would serve as the UI. Interestingly, Piper doesn’t use svg merely for display. It uses specific attributes to identify the location where to put clickable buttons on the UI, or layers to choose which leads to show on each screen, and which ones to highlight on mouse hover.

After I successfully implemented all of the features I considered necessary, I decided to open PRs to Piper and libratbag. After a protracted (by me) review process, both PRs were successfully merged, and will be available in the following release, likely to be v0.18.

Final thoughts

Even though this project was a success, I feel a slight bittersweet aftertaste. I successfully managed to implement everything that I wanted, but what is the likelihood that someone will be using this 8 year-old mouse nowadays, and on linux on top of that? Even I myself don’t use the tool that often.

I am at least glad to be able to have a free and open implementation of the protocol for posterity’s sake. At the very least, I hope this post encourages anyone to try the same for their own mice or any other device, either with the specific goal of re-implementing the configuration protocol, or simply exploring the possibilities that the USB protocol can offer.

And finally, this is a great example as to how free software benefits everyone. I benefit because I can run and modify the program without any restrictions. The project benefits from the contributions, and other users benefit from the collaboration and implementation of new features. This is just a small part of why I love free software.

The Wireshark USB dissector seems to not be fully implemented. In particular, the USBHID component does not seem to be parsing the GET_REPORT Response, so the response data does not get dissected and displayed neatly. Given this, I attempted to implement my own dissector that would parse the responses into a proper USBHID response packet. I got fairly deep into implementing this, but the usefulness was dwarfed by the effort of implementing and using it, so I did not proceed any further. ↩︎
I intentionally chose to forgo reverse engineering the macros. I had never used them, nor was planning to, and it would have added a significant amount of additional work to this process. ↩︎

Fishing Fantasy, or how to fix a 15 year old game-breaking bug

Sun, 12 Mar 2023 00:00:00 +0000

Introduction: Fishing Fantasy

There are videogames that can make you reexperience it as if it was the first time. For me, one of these is Tomb Raider 3. It is the first videogame I remember playing (or trying to). I am using the word “playing” a bit loosely here, since I must have been no older than 3, sitting on my dad’s lap and rage quitting after the first encounter with an enemy. I could hardly move the character, let alone aim and defeat even the monkeys in the starting area. It was nevertheless very exciting to watch my dad play. The brightest memory I have is watching my dad fight a three-headed dragon in Trajan’s Markets while jumping left to right and avoiding deadly fireballs.

Another game that sits very close to my heart is the main topic of this article. This game is called “Fishing Fantasy: Buzzrod”, which came out in 2005 for the PS2. It is a cartoonish looking RPG fishing game with focus on exploration and experimentation. You play as Patt Possom, a possum archaeologist striving to be like his renowned archaeologist uncle. While looking through his uncle’s books, he finds a note describing an ancient civilization which worshipped fish as messengers of the gods. He decides to go on an expedition to this remote region and figure out the secrets of the ancients.

The mechanics of this game are simple. You need to catch fish to obtain items to craft lures with which to catch more fish. The game is divided in levels, each one with a specific goal of catching certain types or amounts of fish. Each fish only spawns at specific locations and times of day, and can only be caught with specific lures. And the game does not make it easy on you. It does not give any hint as to their possible location and time of day. My father and I kept a journal with all of our findings, hand-drawn maps of the locations, and which lures can be used to catch each fish.

The unskippable level

There is a big catch with this game: The last level is impossible to finish. The level’s objective is to catch the legendary Skullcanth fish. This fish can only be caught with the Buzz lure, but to craft it you need the following:

2× Mysterious Lithograph 1
2× Mysterious Lithograph 2
3× Mysterious Lithograph 3
1× Big Fallen Leaf
1× Honey

These lithographs are given after completing each level, of which there are 6 before the final stage, but the lure requires 7 of them. Therefore, this lure is impossible to craft, making Skullcanth impossible to catch and making the level impossible to complete.

Recipe menu with Buzz selected, showing one missing crafting item

We must have spent weeks with my dad trying to figure out where to get the missing lithograph, revisiting all levels, catching each fish dozens of times, but in the end we gave up. 15 years later, I decided to replay this game on an emulator for nostalgia’s sake. When faced with the same bug, I chose not to despair, to go further and investigate.

Reading hex and finding the buggy byte

When faced with such a task, your first instinct should be to search online for similar experiences. Due to not being an incredibly popular game, there were very few hits. Most of them were complaining about this exact same issue. There was exactly one comment that mentioned overcoming this bug by modifying memory card data. This was my first attempt, but making sense of memory card data layout proved too hard. I found out after the fact that my emulator was compressing the saves in the virtual memory card, the likely culprit of my confusion when analyzing the save games.

Another avenue that opened up was trying to modify the game binary itself. This is evidently not possible in a real PS2, but given modern emulator technology, it becomes a trivial task. As such I was ready for diving in with Bless into the hexadecimal ocean to find my legendary bugfix.

Going in I had the goal of finding the location where the recipe data is located. I assumed that all of the properties for a given lure would be defined together, including its name, properties, such as taste, light or smell, as well as its recipe. The lure that we need to craft is called Buzz, so I searched for that straight away. The name of the lure is a bit unfortunate, given that the game is called Buzzrod, which produced hundreds of erroneous hits.

After skipping the first few results, I seemed to find my objective. A region of the binary with the names and descriptions of lures separated by a bunch of binary data. It was not immediately clear where each lure ended and the next one began. For this I scrolled up to the first entry of this list to find the pattern which the list follows. It appeared that each entry in the list started with the byte pattern 0xF8FFFF7F, after which followed the name of the lure. This pattern seemed to hold at the end of the list, so I took it as confirmation.

The objective now was to find the recipe within this 368 byte long region of memory. We know that the recipe requires 2 items of a type, 2 more of another type, 3 more of another, plus 1 more of 2 different types. The task proved easy. The sequence 26 01 26 01 27 01 27 01 28 01 28 01 28 01 2B 01 2C 01 immediately lit up. We can deduce that this lure requires the following:

2× Item(id=0x26)
2× Item(id=0x27)
3× Item(id=0x28)
1× Item(id=0x2B)
1× Item(id=0x2C)

We need to modify this sequence so that we only require 2 items of id 0x28, while not breaking address references. As easy as deleting one of the 28 01 and appending 00 00 at the end of the recipe, right after 2C 01. If all you came here for is to fix your .bin file, you can stop reading now.

We are now ready to boot up the game again and see what happens. We head straight to the lure crafting menu, to find that our lure is ready to craft!

Crafting the lure Buzz

With our new lure at our disposal, we set out to catch the legendary Skullcanth. Since a picture is worth a thousand words, the following 4:30 minute long video at 30 fps is worth over 8 million words. Enjoy:

Before sending us back to the start menu, the game gives us a (not so) cryptic message.

Congratulations.

Something good might happen if you save the current state.

Sounds very intriguing. What happens if we load this new save?

Beyond Skullcanth

After loading this new save game, a new cutscene is shown, and we find ourselves in The Lost Ruins, the main lobby and warp hub of the game.

The very first thing that we notice is that the warp gates, which used to be pale blue, turned into a lightsaber red color. It doesn’t take long to find out that we’ve been rewarded with the monumental task of catching the legendary fish of each stage. What follows is, to my knowledge, a playlist of never seen before legendary fishes being caught.

After catching Opa, the last legendary fish, we are rewarded with one last cutscene, altogether with the final progress screen. The game suggests again that something good might happen if we save the current state. We load the new save again and are greeted by a new cutscene. However, this is the end of the novelty, since we are sent back to The Lost Ruins with the same objective we had earlier: Catch every legendary fish. We can take this as an opportunity for collecting any leftover items or crafting the last few lures, as well as experimenting with the different lures and fish.

Making sense of hex data

My desire to fully understand the game did not stop here though. I had the full binary description of all the lures, but it was mostly unintelligible. It is likely that the same kind of data exists for the different fish and items. Might these values hold the key to understanding the affinity of each fish for specific lures? What else could I uncover in these values?

The task of finding the properties for the fish and items proved as easy as it it was to find the lures. A quick search by name and I easily found the range of bytes that corresponded to their properties. After I found these binary ranges I developed a small script in python to split them and analyze the contents with the struct package. Some fields were mostly self-explanatory. Each item’s name and description were encoded in plain ASCII as a fixed length string. We also know where to find the recipe. Apart from these, though, the rest looks like random bytes.

After exploring the data a bit more using Bless, we can already identify certain patterns in some values. For example, the values between 0x3F and 0x41 can signify a floating point number, such as CC CC 4C 3F, which represents (in little endian) the value 0.8, while 00 00 70 41 is equal to 15.0. We won’t get much further than that by simply examining the structure, though. We have to put on our experimentation hats, modify some values and see how they affect the behavior of each item in the game. With this procedure, I managed to decypher the greatest majority of values, such as fish lure preferences or item drops, as well as moderately fun facts and images, such as the weight range being encoded in hectograms, or a tiny but very ferocious Red Plate.

I could have called it quits here, but I was in the process of creating a wikia for this game, and there was one critical thing missing: Item icons. I had to get the original icons used for each of the in-game items. I obviously searched online for any traces of these assets, but all I found was the original website of the game developer. There are some item icons in there; unfortunately, not all of them.

I decided to jump back into the binary to find them. Luckily, the binary image was in .iso format, which can be easily extracted into several files and folders. That’s where my luck ran out. All I was left with is a bunch of files with unknown extensions, such as .BD, .SD, .PAC, etc, and no way of opening them. After looking through the generated files, I concluded that the image items must be stored in the file named CDROM.PAC for two reasons. Firstly, it was the biggest file in the archive, and secondly, the rest of the files were adequately grouped in folders with meaningful names, and none of them referenced any image, picture, or texture keywords.

I, as one would expect, opened this file in a hex editor, and what I found would not surprise you. It’s an archive file. It started with the magic bytes PACK and a list of filenames. I searched online for a long while, but unfortunately I found nothing resembling it, nor anything that claimed to open this file format.

Writing your own archiving format

Again, I was left with no choice but to write my own archiving utility. Surprisingly and luckily for me, I’m not the first lunatic to have had this need. I found this little project called QuickBMS. QuickBMS is a GPLv2 licensed “universal script based files extractor and reimporter”. It has its own domain specific language to describe the structure of the archiving format, including endianness and byte alignment, which it uses to unpack a given file archive.

After several trial and error iterations, I managed to write a script that made QuickBMS produce a couple thousand file-looking things on my folder. Most of these files had fairly cryptic names consisting of 3 letters followed by 4 numbers, such as CDB_0120.TXL. A few of them, though, were very understandable. Some of them even familiar, such as lure.info and fish.info, which contained the information we analyzed earlier.

After perusing this catalog and inspecting a few files, I realized that all files with the extension .TXL were archives with the exact same format as CDROM.PAC. One of these files, coincidentally, was called ITEM.TXL. Could it contain my precious item icons? I ran the QuickBMS script against all of these files and what I found inside left me with mixed feelings. On the one hand, I was definitely getting closer to my goal. Inside ITEM.TXL lay a list of files titled Mat01.tm2, clearly referencing the lure construction materials within the game. On the other hand, I had no clue what the extension .tm2 could possibly mean.

Searching online led me to conclude that this is a propietary format called TIM2 created by Sony for the PlayStation 2. Unfortunately, this format is not widely used outside of its designated purpose, so the tooling is scarce. Searching for programs that could open such file format showed only one: XnViewMP, a freeware image organizer. Why that was the only known program that can handle these images we’ll never know. I proceeded to install this program and it did indeed open these files without a hitch. It only had some trouble with transparencies, but that can be understood given the rarity of this format.

I could have stopped there. Simply load each image into the program, export them as pngs, fiddle around with transparencies and be done with it. I obviously didn’t stop there.

Implementing a new format in ImageMagick

I set out to fully understand this file format. How can such a format be almost forgotten about? I searched online for a specification of the format and, to my surprise, I actually found some interesting stuff. I found a fairly complete specification of the format in a wiki called XeNTaX, dedicated to preserving game file formats, as well as another fairly complete spec in another wiki called “The Cutting Room Floor”, dedicated to unearthing and researching unused and cut content from videogames. It feels encouraging to see that I’m not the only one that took the time to uncover some secrets and preserve some history. Plus, what would the internet be without these hyper-specific communities? I also found a very detailed post in japanese and a forum post in russian with some more details, plus some more webpages which I’ll leave below in the references.¹ ²

But that was not all that I found. It would appear that somebody leaked some kind of internal tool on github that was used for loading and handling TIM2 images. It contains a copyright notice for “2002 Sony Computer Entertainment Inc.”, and the files are prefixed with “SCE CONFIDENTIAL”. The comments, once converted to EUC-JP encoding, seem to be coherent and insightful, which leads me to believe that it might be Sony’s original source code. Nevertheless, I chose to proceed without the aforementioned leaked specification for one simple reason: it would be copyright infringement.

After reading and re-reading all these resources many times, I began to have some idea as to how this image format was designed. Luckily for me, it was a fairly simple format, without any compression like png or jpeg.

This format uses a technique called indexed color. The binary is divided into two parts. Firstly, the CLUT (Color LookUp Table) with the palette of the image, and secondly, the list of pixels, each referencing one of the entries in this CLUT. Additionally, the CLUT can be stored in two different CSM (CLUT Storage Mode). On CSM1 the CLUT is shuffled, so it needs to be deshuffled. The reason it is shuffled appears to be related to memory bus layout of the PS2³. The long and short of it is that the 2nd and 3rd block of each page in the CLUT must be swapped.

After gathering all of this documentation, I set out to write my own decoder for the TIM2 format in ImageMagick. I quickly found out that ImageMagick already supports the TIM image format, the predecessor of TIM2, so I used it as a template for implementing my own. After many hours of figuring out how the ImageMagick API worked, I managed to produce the following image:

That’s it! I managed to extract the image icons that I was looking for! I immediately uploaded them into the wiki I was building. At the same time, I opened a PR for ImageMagick to include my TIM2 image decoder. It took a couple of weeks for my PR to be accepted, and it was finally included in ImageMagick 7.0.8. It makes me proud to say that running magick identify -list format now shows TM2* TIM2 r-- PS2 TIM2 among the supported formats.

Addendum

It seems I may not have been the first to post proof of fixing this bug. I later found out that someone posted videos of some of the Legendary fish being caught and even finishing the game on YouTube. The author mentions on the comments using a modified save game they found online, so I can still take credit for being the first to document how to fix the actual bug, instead of using external workarounds.

Additionally, I found out there exists a version of the game made specifically for the japanese market, titled “BuzzRod: Fishing Fantasy”. I found out that this version of the game is not bugged, as shown in a video by a japanese youtuber. Either it isn’t bugged, or he fixed it as well, which sounds unlikely.

Regarding the TIM2 encoder, there are still some weird issues about it. When I was implementing it, I was exclusively trying to convert from .tm2 to .png, as that was the format I wanted to use in the wiki. I later found out that for some odd reason, converting to other formats does not work as well as it does for .png. For example, converting to .gif does not preserve the transparency, and converting to .jpg produces an almost completely black image.