[chirp_devel] The End of Days - #495

[Dan Smith]

Hi all,

I dunno why, but I recently started working on some python3 conversion stuff for chirp. In hindsight, I don't know what would have triggered such a masochistic act. If you are interested in such things, read on. If not, save yourself.

I've been putting this off for a long time for a variety of reasons. First, python3 doesn't buy us much (which is generally the reaction by most people that are having to convert), and so it's a lot of work for seemingly little gain. Python3 is a nice language, but it's different from Python 2, and moving from one to the other isn't really a benefit on its own. A lot of projects will never convert, and I kinda thought (think?) chirp would be in that bucket. The python community itself is not supporting python2.7 beyond 2020, although many vendors will have to do it themselves for people that won't or haven't moved. For chirp, the big "problem" is the linux distros (and to some extend MacOS via homebrew) who are dropping support for python2. Even if it's still installable separately, things like pip, tox, etc are all going to move on at some point.

So, I've been trying to figure out what we would have to do to support it. Our UI uses pygtk, which does not (and will not) work on python3. I really thought that was going to be the major sticking point for us, as I'm pretty much the only one that works on the UI, and it's close to the bottom of my list of "fun things to do." However, it turns out that the strict unicode/bytes enforcement nature of python3 is a lot more problematic than I thought it was going to be. The actual enforcement is straightforward and it's easy to write new code that does it properly. Having to learn how to make python3 happy about unicode and bytes for other projects really helped me understand unicode, so there's some benefit to it in general. However, we have a ton of drivers that encode binary in strings, which does not work the same way in python3. Not only is it a major undertaking to convert all of those drivers just on sheer lines of code, but it's fairly error prone and in need of testing, which isn't something that is easy to do. We don't automate it for obvious reasons, and many people that write a driver move on, sell their radio, or were just borrowing one to write the driver. So, if someone converts the code from strings to bytes, it's hard to know if it's right until someone with access to that radio can do an extensive test.

Anyway, I've spent a bunch of time on it and I have some things working. Some of it is a real mess and I've spent a lot of time yelling at the screen. This is basically what I have working:

 - You can run chirpw under python3 and it starts
 - Drivers that aren't even syntax-compatible with python3 will gracefully (albeit noisily) not prevent
   importing the chirp modules
 - You can run the unit tests under python3 (tox -epy3)
 - You can run the driver tests that work (and are importable) under the same tox target
 - You can clone to/from an Icom ID-800 (and likely some other Icom models)
 - You can use most of the basic UI to some degree

A lot of drivers need 2to3 conversion just so they will be importable, and I haven't done that yet. I plan to do them slowly as I run tests and fix up harder issues. After all the drivers at least import, I'd like to work on making some radios fully byte-clean to make sure the core can support both (so that new drivers can be written properly) and to provide a template for moving forward with the rest.

The big parts of the underlying core that need to change that impact drivers are (unsurprisingly) MemoryMap, bitwise, and the serial stuff. Because nearly every driver does a lot of '\xFF' type stuff in strings, interacts with MemoryMap and Serial in those ways, I've implemented some compatibility layers. I really want(ed) to convert the core to be all unicode/bytes clean, but that breaks every single driver straight off. Since I don't think we'll ever make progress if we have to touch every single driver before we can get it in the hands of real users, I've taken the approach of trying to make the underlying stuff clean, but wrap it with layers to make existing drivers work. As a driver gets converted, we can ideally hand it non-wrapped byte-clean versions of the core objects.

So, MemoryMapBytes is a byte-clean version of MemoryMap, with the latter being a wrapper around it to make it work as the drivers expect with just strings. Similarly, CompatSerial wraps Serial, converting back and forth between bytes and strings in a way that will make the drivers mostly work.

With this stuff underneath, I was able to use an ID-800 I have sitting around with chirp under python3 with minimal fuss. The changes to the tree are non-trivial, and far from complete obviously. The diffstat so far is:

  52 files changed, 1149 insertions(+), 508 deletions(-)

...and remember, that's basically no driver changes, with only a few that actually import by luck currently.

The changes are currently in the "py3" branch of the tree, so check that out if you want to have a look:

  http://d-rats.com/hg/hgwebdir.cgi/chirp.hg/shortlog/7b4ced760655

I think soon it will be worth getting something integrated into the build and into the hands of linux users (even if just in tarball form) to get some help testing. I haven't tried building it on windows, even against python2. I've been trying to keep the tests passing in python2 as I go (which has been a challenge). Ideally we'd merge this with the main repo at some point, but I'm not confident enough in its sanity to do that yet.

Is anyone interested in helping with this? Several people have showed up here in the past looking to do it themselves or help, but nothing has ever materialized. I'm hoping that this will get the ball rolling. It would be really helpful to have some people with the ability to run this on linux under python3 to test every radio they have against the current driver, and either report or fix what they find.

And, as a bonus for reading (or skimming) this far, I leave you with the big reveal:

  https://i.imgur.com/ZHOnPe3.png

--Dan