[chirp_devel] The End of Days - #495

[Dan Clemmensen]

I would like to help with the Python 3 effort, but I'm still a newbie. I
think I'm within a week of submitting my new ft4.py driver, and I would
like to at least make sure it imports in Python 3. Is there a trivial way
to do this? Do I need to do a full-up download of the "py3" branch? If I do
the full download, Can I use your ID-800 driver as an example of the
changes to make?

Because the FT-4 is (apparently) completely different than all
earlier-generation Yeasu HTs, my driver has very little in common with any
other driver. It passes the unit test and I supports all known features of
the FT-4, but does not yet support the the FT-65.

I use Linux (specifically, Gentoo) so I have a dual Python 2-Python 3
environment.

For testing: I know we really need to test with real radios, but almost all
testing can be done using image files first, right?
In addition, I suspect most radios' serial port comms can be emulated.
Since a lot of radios share protocols even when the actual images are very
different, the number of emulators would be a good deal smaller than the
number of drivers. For example tthe FT-4 serial protocol is identical to
the th9000.py and anytone_ht.py protocols event though the images bear no
apparent relationship to each other. If you wish I can write an emulator
for this protocol. The test suite would use serial pseudo-ports to connect
the emulator to CHIRP, and we would need a simple config system to pick the
proper emulator for each type of image file.  (I may be a python/CHIRP/Ham
newbie, but I've been a data comms developer for 40 years.)

On Fri, Feb 8, 2019 at 3:49 PM Dan Smith via chirp_devel <
chirp_devel at intrepid.danplanet.com> wrote:

> Hi all,
>
> I dunno why, but I recently started working on some python3 conversion
> stuff for chirp. In hindsight, I don't know what would have triggered such
> a masochistic act. If you are interested in such things, read on. If not,
> save yourself.
>
> I've been putting this off for a long time for a variety of reasons.
> First, python3 doesn't buy us much (which is generally the reaction by most
> people that are having to convert), and so it's a lot of work for seemingly
> little gain. Python3 is a nice language, but it's different from Python 2,
> and moving from one to the other isn't really a benefit on its own. A lot
> of projects will never convert, and I kinda thought (think?) chirp would be
> in that bucket. The python community itself is not supporting python2.7
> beyond 2020, although many vendors will have to do it themselves for people
> that won't or haven't moved. For chirp, the big "problem" is the linux
> distros (and to some extend MacOS via homebrew) who are dropping support
> for python2. Even if it's still installable separately, things like pip,
> tox, etc are all going to move on at some point.
>
> So, I've been trying to figure out what we would have to do to support it.
> Our UI uses pygtk, which does not (and will not) work on python3. I really
> thought that was going to be the major sticking point for us, as I'm pretty
> much the only one that works on the UI, and it's close to the bottom of my
> list of "fun things to do." However, it turns out that the strict
> unicode/bytes enforcement nature of python3 is a lot more problematic than
> I thought it was going to be. The actual enforcement is straightforward and
> it's easy to write new code that does it properly. Having to learn how to
> make python3 happy about unicode and bytes for other projects really helped
> me understand unicode, so there's some benefit to it in general. However,
> we have a ton of drivers that encode binary in strings, which does not work
> the same way in python3. Not only is it a major undertaking to convert all
> of those drivers just on sheer lines of code, but it's fairly error prone
> and in need of testing,
>   which isn't something that is easy to do. We don't automate it for
> obvious reasons, and many people that write a driver move on, sell their
> radio, or were just borrowing one to write the driver. So, if someone
> converts the code from strings to bytes, it's hard to know if it's right
> until someone with access to that radio can do an extensive test.
>
> Anyway, I've spent a bunch of time on it and I have some things working.
> Some of it is a real mess and I've spent a lot of time yelling at the
> screen. This is basically what I have working:
>
>  - You can run chirpw under python3 and it starts
>  - Drivers that aren't even syntax-compatible with python3 will gracefully
> (albeit noisily) not prevent
>    importing the chirp modules
>  - You can run the unit tests under python3 (tox -epy3)
>  - You can run the driver tests that work (and are importable) under the
> same tox target
>  - You can clone to/from an Icom ID-800 (and likely some other Icom models)
>  - You can use most of the basic UI to some degree
>
> A lot of drivers need 2to3 conversion just so they will be importable, and
> I haven't done that yet. I plan to do them slowly as I run tests and fix up
> harder issues. After all the drivers at least import, I'd like to work on
> making some radios fully byte-clean to make sure the core can support both
> (so that new drivers can be written properly) and to provide a template for
> moving forward with the rest.
>
> The big parts of the underlying core that need to change that impact
> drivers are (unsurprisingly) MemoryMap, bitwise, and the serial stuff.
> Because nearly every driver does a lot of '\xFF' type stuff in strings,
> interacts with MemoryMap and Serial in those ways, I've implemented some
> compatibility layers. I really want(ed) to convert the core to be all
> unicode/bytes clean, but that breaks every single driver straight off.
> Since I don't think we'll ever make progress if we have to touch every
> single driver before we can get it in the hands of real users, I've taken
> the approach of trying to make the underlying stuff clean, but wrap it with
> layers to make existing drivers work. As a driver gets converted, we can
> ideally hand it non-wrapped byte-clean versions of the core objects.
>
> So, MemoryMapBytes is a byte-clean version of MemoryMap, with the latter
> being a wrapper around it to make it work as the drivers expect with just
> strings. Similarly, CompatSerial wraps Serial, converting back and forth
> between bytes and strings in a way that will make the drivers mostly work.
>
> With this stuff underneath, I was able to use an ID-800 I have sitting
> around with chirp under python3 with minimal fuss. The changes to the tree
> are non-trivial, and far from complete obviously. The diffstat so far is:
>
>   52 files changed, 1149 insertions(+), 508 deletions(-)
>
> ...and remember, that's basically no driver changes, with only a few that
> actually import by luck currently.
>
> The changes are currently in the "py3" branch of the tree, so check that
> out if you want to have a look:
>
>   http://d-rats.com/hg/hgwebdir.cgi/chirp.hg/shortlog/7b4ced760655
>
> I think soon it will be worth getting something integrated into the build
> and into the hands of linux users (even if just in tarball form) to get
> some help testing. I haven't tried building it on windows, even against
> python2. I've been trying to keep the tests passing in python2 as I go
> (which has been a challenge). Ideally we'd merge this with the main repo at
> some point, but I'm not confident enough in its sanity to do that yet.
>
> Is anyone interested in helping with this? Several people have showed up
> here in the past looking to do it themselves or help, but nothing has ever
> materialized. I'm hoping that this will get the ball rolling. It would be
> really helpful to have some people with the ability to run this on linux
> under python3 to test every radio they have against the current driver, and
> either report or fix what they find.
>
> And, as a bonus for reading (or skimming) this far, I leave you with the
> big reveal:
>
>   https://i.imgur.com/ZHOnPe3.png
>
> --Dan
> _______________________________________________
> chirp_devel mailing list
> chirp_devel at intrepid.danplanet.com
> http://intrepid.danplanet.com/mailman/listinfo/chirp_devel
> Developer docs: http://chirp.danplanet.com/projects/chirp/wiki/Developers
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://intrepid.danplanet.com/pipermail/chirp_devel/attachments/20190208/97f5c472/attachment-0001.html