I would like to help with the Python 3 effort, but I'm still a newbie. I think I'm within a week of submitting my new ft4.py driver, and I would like to at least make sure it imports in Python 3. Is there a trivial way to do this? Do I need to do a full-up download of the "py3" branch? If I do the full download, Can I use your ID-800 driver as an example of the changes to make?
Because the FT-4 is (apparently) completely different than all earlier-generation Yeasu HTs, my driver has very little in common with any other driver. It passes the unit test and I supports all known features of the FT-4, but does not yet support the the FT-65.
I use Linux (specifically, Gentoo) so I have a dual Python 2-Python 3 environment.
For testing: I know we really need to test with real radios, but almost all testing can be done using image files first, right? In addition, I suspect most radios' serial port comms can be emulated. Since a lot of radios share protocols even when the actual images are very different, the number of emulators would be a good deal smaller than the number of drivers. For example tthe FT-4 serial protocol is identical to the th9000.py and anytone_ht.py protocols event though the images bear no apparent relationship to each other. If you wish I can write an emulator for this protocol. The test suite would use serial pseudo-ports to connect the emulator to CHIRP, and we would need a simple config system to pick the proper emulator for each type of image file. (I may be a python/CHIRP/Ham newbie, but I've been a data comms developer for 40 years.)
On Fri, Feb 8, 2019 at 3:49 PM Dan Smith via chirp_devel < chirp_devel@intrepid.danplanet.com> wrote:
Hi all,
I dunno why, but I recently started working on some python3 conversion stuff for chirp. In hindsight, I don't know what would have triggered such a masochistic act. If you are interested in such things, read on. If not, save yourself.
I've been putting this off for a long time for a variety of reasons. First, python3 doesn't buy us much (which is generally the reaction by most people that are having to convert), and so it's a lot of work for seemingly little gain. Python3 is a nice language, but it's different from Python 2, and moving from one to the other isn't really a benefit on its own. A lot of projects will never convert, and I kinda thought (think?) chirp would be in that bucket. The python community itself is not supporting python2.7 beyond 2020, although many vendors will have to do it themselves for people that won't or haven't moved. For chirp, the big "problem" is the linux distros (and to some extend MacOS via homebrew) who are dropping support for python2. Even if it's still installable separately, things like pip, tox, etc are all going to move on at some point.
So, I've been trying to figure out what we would have to do to support it. Our UI uses pygtk, which does not (and will not) work on python3. I really thought that was going to be the major sticking point for us, as I'm pretty much the only one that works on the UI, and it's close to the bottom of my list of "fun things to do." However, it turns out that the strict unicode/bytes enforcement nature of python3 is a lot more problematic than I thought it was going to be. The actual enforcement is straightforward and it's easy to write new code that does it properly. Having to learn how to make python3 happy about unicode and bytes for other projects really helped me understand unicode, so there's some benefit to it in general. However, we have a ton of drivers that encode binary in strings, which does not work the same way in python3. Not only is it a major undertaking to convert all of those drivers just on sheer lines of code, but it's fairly error prone and in need of testing, which isn't something that is easy to do. We don't automate it for obvious reasons, and many people that write a driver move on, sell their radio, or were just borrowing one to write the driver. So, if someone converts the code from strings to bytes, it's hard to know if it's right until someone with access to that radio can do an extensive test.
Anyway, I've spent a bunch of time on it and I have some things working. Some of it is a real mess and I've spent a lot of time yelling at the screen. This is basically what I have working:
- You can run chirpw under python3 and it starts
- Drivers that aren't even syntax-compatible with python3 will gracefully
(albeit noisily) not prevent importing the chirp modules
- You can run the unit tests under python3 (tox -epy3)
- You can run the driver tests that work (and are importable) under the
same tox target
- You can clone to/from an Icom ID-800 (and likely some other Icom models)
- You can use most of the basic UI to some degree
A lot of drivers need 2to3 conversion just so they will be importable, and I haven't done that yet. I plan to do them slowly as I run tests and fix up harder issues. After all the drivers at least import, I'd like to work on making some radios fully byte-clean to make sure the core can support both (so that new drivers can be written properly) and to provide a template for moving forward with the rest.
The big parts of the underlying core that need to change that impact drivers are (unsurprisingly) MemoryMap, bitwise, and the serial stuff. Because nearly every driver does a lot of '\xFF' type stuff in strings, interacts with MemoryMap and Serial in those ways, I've implemented some compatibility layers. I really want(ed) to convert the core to be all unicode/bytes clean, but that breaks every single driver straight off. Since I don't think we'll ever make progress if we have to touch every single driver before we can get it in the hands of real users, I've taken the approach of trying to make the underlying stuff clean, but wrap it with layers to make existing drivers work. As a driver gets converted, we can ideally hand it non-wrapped byte-clean versions of the core objects.
So, MemoryMapBytes is a byte-clean version of MemoryMap, with the latter being a wrapper around it to make it work as the drivers expect with just strings. Similarly, CompatSerial wraps Serial, converting back and forth between bytes and strings in a way that will make the drivers mostly work.
With this stuff underneath, I was able to use an ID-800 I have sitting around with chirp under python3 with minimal fuss. The changes to the tree are non-trivial, and far from complete obviously. The diffstat so far is:
52 files changed, 1149 insertions(+), 508 deletions(-)
...and remember, that's basically no driver changes, with only a few that actually import by luck currently.
The changes are currently in the "py3" branch of the tree, so check that out if you want to have a look:
http://d-rats.com/hg/hgwebdir.cgi/chirp.hg/shortlog/7b4ced760655
I think soon it will be worth getting something integrated into the build and into the hands of linux users (even if just in tarball form) to get some help testing. I haven't tried building it on windows, even against python2. I've been trying to keep the tests passing in python2 as I go (which has been a challenge). Ideally we'd merge this with the main repo at some point, but I'm not confident enough in its sanity to do that yet.
Is anyone interested in helping with this? Several people have showed up here in the past looking to do it themselves or help, but nothing has ever materialized. I'm hoping that this will get the ball rolling. It would be really helpful to have some people with the ability to run this on linux under python3 to test every radio they have against the current driver, and either report or fix what they find.
And, as a bonus for reading (or skimming) this far, I leave you with the big reveal:
https://i.imgur.com/ZHOnPe3.png
--Dan _______________________________________________ chirp_devel mailing list chirp_devel@intrepid.danplanet.com http://intrepid.danplanet.com/mailman/listinfo/chirp_devel Developer docs: http://chirp.danplanet.com/projects/chirp/wiki/Developers