[chirp_devel] The End of Days - #495
Hi all,
I dunno why, but I recently started working on some python3 conversion stuff for chirp. In hindsight, I don't know what would have triggered such a masochistic act. If you are interested in such things, read on. If not, save yourself.
I've been putting this off for a long time for a variety of reasons. First, python3 doesn't buy us much (which is generally the reaction by most people that are having to convert), and so it's a lot of work for seemingly little gain. Python3 is a nice language, but it's different from Python 2, and moving from one to the other isn't really a benefit on its own. A lot of projects will never convert, and I kinda thought (think?) chirp would be in that bucket. The python community itself is not supporting python2.7 beyond 2020, although many vendors will have to do it themselves for people that won't or haven't moved. For chirp, the big "problem" is the linux distros (and to some extend MacOS via homebrew) who are dropping support for python2. Even if it's still installable separately, things like pip, tox, etc are all going to move on at some point.
So, I've been trying to figure out what we would have to do to support it. Our UI uses pygtk, which does not (and will not) work on python3. I really thought that was going to be the major sticking point for us, as I'm pretty much the only one that works on the UI, and it's close to the bottom of my list of "fun things to do." However, it turns out that the strict unicode/bytes enforcement nature of python3 is a lot more problematic than I thought it was going to be. The actual enforcement is straightforward and it's easy to write new code that does it properly. Having to learn how to make python3 happy about unicode and bytes for other projects really helped me understand unicode, so there's some benefit to it in general. However, we have a ton of drivers that encode binary in strings, which does not work the same way in python3. Not only is it a major undertaking to convert all of those drivers just on sheer lines of code, but it's fairly error prone and in need of testing, which isn't something that is easy to do. We don't automate it for obvious reasons, and many people that write a driver move on, sell their radio, or were just borrowing one to write the driver. So, if someone converts the code from strings to bytes, it's hard to know if it's right until someone with access to that radio can do an extensive test.
Anyway, I've spent a bunch of time on it and I have some things working. Some of it is a real mess and I've spent a lot of time yelling at the screen. This is basically what I have working:
- You can run chirpw under python3 and it starts - Drivers that aren't even syntax-compatible with python3 will gracefully (albeit noisily) not prevent importing the chirp modules - You can run the unit tests under python3 (tox -epy3) - You can run the driver tests that work (and are importable) under the same tox target - You can clone to/from an Icom ID-800 (and likely some other Icom models) - You can use most of the basic UI to some degree
A lot of drivers need 2to3 conversion just so they will be importable, and I haven't done that yet. I plan to do them slowly as I run tests and fix up harder issues. After all the drivers at least import, I'd like to work on making some radios fully byte-clean to make sure the core can support both (so that new drivers can be written properly) and to provide a template for moving forward with the rest.
The big parts of the underlying core that need to change that impact drivers are (unsurprisingly) MemoryMap, bitwise, and the serial stuff. Because nearly every driver does a lot of '\xFF' type stuff in strings, interacts with MemoryMap and Serial in those ways, I've implemented some compatibility layers. I really want(ed) to convert the core to be all unicode/bytes clean, but that breaks every single driver straight off. Since I don't think we'll ever make progress if we have to touch every single driver before we can get it in the hands of real users, I've taken the approach of trying to make the underlying stuff clean, but wrap it with layers to make existing drivers work. As a driver gets converted, we can ideally hand it non-wrapped byte-clean versions of the core objects.
So, MemoryMapBytes is a byte-clean version of MemoryMap, with the latter being a wrapper around it to make it work as the drivers expect with just strings. Similarly, CompatSerial wraps Serial, converting back and forth between bytes and strings in a way that will make the drivers mostly work.
With this stuff underneath, I was able to use an ID-800 I have sitting around with chirp under python3 with minimal fuss. The changes to the tree are non-trivial, and far from complete obviously. The diffstat so far is:
52 files changed, 1149 insertions(+), 508 deletions(-)
...and remember, that's basically no driver changes, with only a few that actually import by luck currently.
The changes are currently in the "py3" branch of the tree, so check that out if you want to have a look:
http://d-rats.com/hg/hgwebdir.cgi/chirp.hg/shortlog/7b4ced760655
I think soon it will be worth getting something integrated into the build and into the hands of linux users (even if just in tarball form) to get some help testing. I haven't tried building it on windows, even against python2. I've been trying to keep the tests passing in python2 as I go (which has been a challenge). Ideally we'd merge this with the main repo at some point, but I'm not confident enough in its sanity to do that yet.
Is anyone interested in helping with this? Several people have showed up here in the past looking to do it themselves or help, but nothing has ever materialized. I'm hoping that this will get the ball rolling. It would be really helpful to have some people with the ability to run this on linux under python3 to test every radio they have against the current driver, and either report or fix what they find.
And, as a bonus for reading (or skimming) this far, I leave you with the big reveal:
https://i.imgur.com/ZHOnPe3.png
--Dan
I would like to help with the Python 3 effort, but I'm still a newbie. I think I'm within a week of submitting my new ft4.py driver, and I would like to at least make sure it imports in Python 3. Is there a trivial way to do this? Do I need to do a full-up download of the "py3" branch? If I do the full download, Can I use your ID-800 driver as an example of the changes to make?
Because the FT-4 is (apparently) completely different than all earlier-generation Yeasu HTs, my driver has very little in common with any other driver. It passes the unit test and I supports all known features of the FT-4, but does not yet support the the FT-65.
I use Linux (specifically, Gentoo) so I have a dual Python 2-Python 3 environment.
For testing: I know we really need to test with real radios, but almost all testing can be done using image files first, right? In addition, I suspect most radios' serial port comms can be emulated. Since a lot of radios share protocols even when the actual images are very different, the number of emulators would be a good deal smaller than the number of drivers. For example tthe FT-4 serial protocol is identical to the th9000.py and anytone_ht.py protocols event though the images bear no apparent relationship to each other. If you wish I can write an emulator for this protocol. The test suite would use serial pseudo-ports to connect the emulator to CHIRP, and we would need a simple config system to pick the proper emulator for each type of image file. (I may be a python/CHIRP/Ham newbie, but I've been a data comms developer for 40 years.)
On Fri, Feb 8, 2019 at 3:49 PM Dan Smith via chirp_devel < chirp_devel@intrepid.danplanet.com> wrote:
Hi all,
I dunno why, but I recently started working on some python3 conversion stuff for chirp. In hindsight, I don't know what would have triggered such a masochistic act. If you are interested in such things, read on. If not, save yourself.
I've been putting this off for a long time for a variety of reasons. First, python3 doesn't buy us much (which is generally the reaction by most people that are having to convert), and so it's a lot of work for seemingly little gain. Python3 is a nice language, but it's different from Python 2, and moving from one to the other isn't really a benefit on its own. A lot of projects will never convert, and I kinda thought (think?) chirp would be in that bucket. The python community itself is not supporting python2.7 beyond 2020, although many vendors will have to do it themselves for people that won't or haven't moved. For chirp, the big "problem" is the linux distros (and to some extend MacOS via homebrew) who are dropping support for python2. Even if it's still installable separately, things like pip, tox, etc are all going to move on at some point.
So, I've been trying to figure out what we would have to do to support it. Our UI uses pygtk, which does not (and will not) work on python3. I really thought that was going to be the major sticking point for us, as I'm pretty much the only one that works on the UI, and it's close to the bottom of my list of "fun things to do." However, it turns out that the strict unicode/bytes enforcement nature of python3 is a lot more problematic than I thought it was going to be. The actual enforcement is straightforward and it's easy to write new code that does it properly. Having to learn how to make python3 happy about unicode and bytes for other projects really helped me understand unicode, so there's some benefit to it in general. However, we have a ton of drivers that encode binary in strings, which does not work the same way in python3. Not only is it a major undertaking to convert all of those drivers just on sheer lines of code, but it's fairly error prone and in need of testing, which isn't something that is easy to do. We don't automate it for obvious reasons, and many people that write a driver move on, sell their radio, or were just borrowing one to write the driver. So, if someone converts the code from strings to bytes, it's hard to know if it's right until someone with access to that radio can do an extensive test.
Anyway, I've spent a bunch of time on it and I have some things working. Some of it is a real mess and I've spent a lot of time yelling at the screen. This is basically what I have working:
- You can run chirpw under python3 and it starts
- Drivers that aren't even syntax-compatible with python3 will gracefully
(albeit noisily) not prevent importing the chirp modules
- You can run the unit tests under python3 (tox -epy3)
- You can run the driver tests that work (and are importable) under the
same tox target
- You can clone to/from an Icom ID-800 (and likely some other Icom models)
- You can use most of the basic UI to some degree
A lot of drivers need 2to3 conversion just so they will be importable, and I haven't done that yet. I plan to do them slowly as I run tests and fix up harder issues. After all the drivers at least import, I'd like to work on making some radios fully byte-clean to make sure the core can support both (so that new drivers can be written properly) and to provide a template for moving forward with the rest.
The big parts of the underlying core that need to change that impact drivers are (unsurprisingly) MemoryMap, bitwise, and the serial stuff. Because nearly every driver does a lot of '\xFF' type stuff in strings, interacts with MemoryMap and Serial in those ways, I've implemented some compatibility layers. I really want(ed) to convert the core to be all unicode/bytes clean, but that breaks every single driver straight off. Since I don't think we'll ever make progress if we have to touch every single driver before we can get it in the hands of real users, I've taken the approach of trying to make the underlying stuff clean, but wrap it with layers to make existing drivers work. As a driver gets converted, we can ideally hand it non-wrapped byte-clean versions of the core objects.
So, MemoryMapBytes is a byte-clean version of MemoryMap, with the latter being a wrapper around it to make it work as the drivers expect with just strings. Similarly, CompatSerial wraps Serial, converting back and forth between bytes and strings in a way that will make the drivers mostly work.
With this stuff underneath, I was able to use an ID-800 I have sitting around with chirp under python3 with minimal fuss. The changes to the tree are non-trivial, and far from complete obviously. The diffstat so far is:
52 files changed, 1149 insertions(+), 508 deletions(-)
...and remember, that's basically no driver changes, with only a few that actually import by luck currently.
The changes are currently in the "py3" branch of the tree, so check that out if you want to have a look:
http://d-rats.com/hg/hgwebdir.cgi/chirp.hg/shortlog/7b4ced760655
I think soon it will be worth getting something integrated into the build and into the hands of linux users (even if just in tarball form) to get some help testing. I haven't tried building it on windows, even against python2. I've been trying to keep the tests passing in python2 as I go (which has been a challenge). Ideally we'd merge this with the main repo at some point, but I'm not confident enough in its sanity to do that yet.
Is anyone interested in helping with this? Several people have showed up here in the past looking to do it themselves or help, but nothing has ever materialized. I'm hoping that this will get the ball rolling. It would be really helpful to have some people with the ability to run this on linux under python3 to test every radio they have against the current driver, and either report or fix what they find.
And, as a bonus for reading (or skimming) this far, I leave you with the big reveal:
https://i.imgur.com/ZHOnPe3.png
--Dan _______________________________________________ chirp_devel mailing list chirp_devel@intrepid.danplanet.com http://intrepid.danplanet.com/mailman/listinfo/chirp_devel Developer docs: http://chirp.danplanet.com/projects/chirp/wiki/Developers
I would like to help with the Python 3 effort, but I'm still a newbie. I think I'm within a week of submitting my new ft4.py driver, and I would like to at least make sure it imports in Python 3. Is there a trivial way to do this? Do I need to do a full-up download of the "py3" branch? If I do the full download, Can I use your ID-800 driver as an example of the changes to make?
Ah, sweet, this will be a good exercise!
Right now, the ID800 driver on the py3 branch works via all the emulation I've got in place. I will try to convert it to be byte-native in the next couple of days and then it will be a reasonable model to follow. All icom radios use the same cloning protocol, so all of that is confined to the icf.py file, so the changes will be spread across that and the actual driver file. That may or may not be similar for yours depending on how you have done it, but the principles will be the same.
Because the FT-4 is (apparently) completely different than all earlier-generation Yeasu HTs, my driver has very little in common with any other driver. It passes the unit test and I supports all known features of the FT-4, but does not yet support the the FT-65.
Cool, you may win the distinction of landing the first py3 native driver in CHIRP :)
I use Linux (specifically, Gentoo) so I have a dual Python 2-Python 3 environment.
For testing: I know we really need to test with real radios, but almost all testing can be done using image files first, right? In addition, I suspect most radios' serial port comms can be emulated. Since a lot of radios share protocols even when the actual images are very different, the number of emulators would be a good deal smaller than the number of drivers. For example tthe FT-4 serial protocol is identical to the th9000.py and anytone_ht.py protocols event though the images bear no apparent relationship to each other. If you wish I can write an emulator for this protocol. The test suite would use serial pseudo-ports to connect the emulator to CHIRP, and we would need a simple config system to pick the proper emulator for each type of image file. (I may be a python/CHIRP/Ham newbie, but I've been a data comms developer for 40 years.)
Yeah, we could write simulators for each of the drivers (or the subset of *protocols*), and those would likely help us in this particular case. However, many (many) radios are insanely sensitive to the timing of the conversation, which makes a simulator not that useful for actually proving that it *actually* works. There are some dumb tests in run_tests.py that simulate a broken serial connection for the purposes of testing the way drivers handle error conditions while cloning, which is a simulator of sorts. We could do a similar thing to test a py3 byte conversion on a model, but that's still a lot of work.
--Dan
Amazing work! I actually played with the codebase a little bit, and implemented basic PyGObject port on Python 2 (the app starts, and basic memory editing functionality is intact). I think Python 3 + PyGObject will be the way to go, but porting to PyGObject will gain us more than the 2to3 conversion.
Zhaofeng Li
On Fri, Feb 8, 2019 at 5:37 PM Dan Smith via chirp_devel chirp_devel@intrepid.danplanet.com wrote:
I would like to help with the Python 3 effort, but I'm still a newbie. I think I'm within a week of submitting my new ft4.py driver, and I would like to at least make sure it imports in Python 3. Is there a trivial way to do this? Do I need to do a full-up download of the "py3" branch? If I do the full download, Can I use your ID-800 driver as an example of the changes to make?
Ah, sweet, this will be a good exercise!
Right now, the ID800 driver on the py3 branch works via all the emulation I've got in place. I will try to convert it to be byte-native in the next couple of days and then it will be a reasonable model to follow. All icom radios use the same cloning protocol, so all of that is confined to the icf.py file, so the changes will be spread across that and the actual driver file. That may or may not be similar for yours depending on how you have done it, but the principles will be the same.
Because the FT-4 is (apparently) completely different than all earlier-generation Yeasu HTs, my driver has very little in common with any other driver. It passes the unit test and I supports all known features of the FT-4, but does not yet support the the FT-65.
Cool, you may win the distinction of landing the first py3 native driver in CHIRP :)
I use Linux (specifically, Gentoo) so I have a dual Python 2-Python 3 environment.
For testing: I know we really need to test with real radios, but almost all testing can be done using image files first, right? In addition, I suspect most radios' serial port comms can be emulated. Since a lot of radios share protocols even when the actual images are very different, the number of emulators would be a good deal smaller than the number of drivers. For example tthe FT-4 serial protocol is identical to the th9000.py and anytone_ht.py protocols event though the images bear no apparent relationship to each other. If you wish I can write an emulator for this protocol. The test suite would use serial pseudo-ports to connect the emulator to CHIRP, and we would need a simple config system to pick the proper emulator for each type of image file. (I may be a python/CHIRP/Ham newbie, but I've been a data comms developer for 40 years.)
Yeah, we could write simulators for each of the drivers (or the subset of *protocols*), and those would likely help us in this particular case. However, many (many) radios are insanely sensitive to the timing of the conversation, which makes a simulator not that useful for actually proving that it *actually* works. There are some dumb tests in run_tests.py that simulate a broken serial connection for the purposes of testing the way drivers handle error conditions while cloning, which is a simulator of sorts. We could do a similar thing to test a py3 byte conversion on a model, but that's still a lot of work.
--Dan
chirp_devel mailing list chirp_devel@intrepid.danplanet.com http://intrepid.danplanet.com/mailman/listinfo/chirp_devel Developer docs: http://chirp.danplanet.com/projects/chirp/wiki/Developers
I still have most of the code for PyGObject. All of the UI works (tested quite thoroughly), but populating the list is a LOT slower, which I've been stuck on for a while and wanted to fix before putting it up. I'll clean it up a bit and drop the current version somewhere tonight, but if we want to use it, it'll take some effort to merge it with current work.
Jake
On Fri, Feb 8, 2019, 8:49 PM Zhaofeng Li via chirp_devel < chirp_devel@intrepid.danplanet.com wrote:
Amazing work! I actually played with the codebase a little bit, and implemented basic PyGObject port on Python 2 (the app starts, and basic memory editing functionality is intact). I think Python 3 + PyGObject will be the way to go, but porting to PyGObject will gain us more than the 2to3 conversion.
Zhaofeng Li
On Fri, Feb 8, 2019 at 5:37 PM Dan Smith via chirp_devel chirp_devel@intrepid.danplanet.com wrote:
I would like to help with the Python 3 effort, but I'm still a newbie.
I think I'm within a week of submitting my new ft4.py driver, and I would like to at least make sure it imports in Python 3. Is there a trivial way to do this? Do I need to do a full-up download of the "py3" branch? If I do the full download, Can I use your ID-800 driver as an example of the changes to make?
Ah, sweet, this will be a good exercise!
Right now, the ID800 driver on the py3 branch works via all the
emulation I've got in place. I will try to convert it to be byte-native in the next couple of days and then it will be a reasonable model to follow. All icom radios use the same cloning protocol, so all of that is confined to the icf.py file, so the changes will be spread across that and the actual driver file. That may or may not be similar for yours depending on how you have done it, but the principles will be the same.
Because the FT-4 is (apparently) completely different than all
earlier-generation Yeasu HTs, my driver has very little in common with any other driver. It passes the unit test and I supports all known features of the FT-4, but does not yet support the the FT-65.
Cool, you may win the distinction of landing the first py3 native driver
in CHIRP :)
I use Linux (specifically, Gentoo) so I have a dual Python 2-Python 3
environment.
For testing: I know we really need to test with real radios, but
almost all testing can be done using image files first, right?
In addition, I suspect most radios' serial port comms can be emulated.
Since a lot of radios share protocols even when the actual images are very different, the number of emulators would be a good deal smaller than the number of drivers. For example tthe FT-4 serial protocol is identical to the th9000.py and anytone_ht.py protocols event though the images bear no apparent relationship to each other. If you wish I can write an emulator for this protocol. The test suite would use serial pseudo-ports to connect the emulator to CHIRP, and we would need a simple config system to pick the proper emulator for each type of image file. (I may be a python/CHIRP/Ham newbie, but I've been a data comms developer for 40 years.)
Yeah, we could write simulators for each of the drivers (or the subset
of *protocols*), and those would likely help us in this particular case. However, many (many) radios are insanely sensitive to the timing of the conversation, which makes a simulator not that useful for actually proving that it *actually* works. There are some dumb tests in run_tests.py that simulate a broken serial connection for the purposes of testing the way drivers handle error conditions while cloning, which is a simulator of sorts. We could do a similar thing to test a py3 byte conversion on a model, but that's still a lot of work.
--Dan
chirp_devel mailing list chirp_devel@intrepid.danplanet.com http://intrepid.danplanet.com/mailman/listinfo/chirp_devel Developer docs:
http://chirp.danplanet.com/projects/chirp/wiki/Developers _______________________________________________ chirp_devel mailing list chirp_devel@intrepid.danplanet.com http://intrepid.danplanet.com/mailman/listinfo/chirp_devel Developer docs: http://chirp.danplanet.com/projects/chirp/wiki/Developers
I still have most of the code for PyGObject. All of the UI works (tested quite thoroughly), but populating the list is a LOT slower, which I've been stuck on for a while and wanted to fix before putting it up. I'll clean it up a bit and drop the current version somewhere tonight, but if we want to use it, it'll take some effort to merge it with current work.
That might be interesting to look at. There is also a giant monolithic patch in #6363 that has some pygi changes, but it pretty much doesn't work at all for me and seems like it might just be the result of running multiple autoconversion tools on the tree.
What I have in the py3 branch does operate against PyGObject, using the pygtk compatibility layer. It's far from perfect, but it's close enough that I feel like it's a good option for the time being.
If we get to the point of being able to run under py3 as well as py2, then converting to all native PyGObject would be good of course.
--Dan
Hi,
After some struggle, I managed to port the Baofeng UV-B5 and TYT TH-350 (subclass of UV-B5) drivers to Python 3, along with various other fixes to the codebase. Attached is the patch.
I've tested uploading, downloading and saving with the TH-350 driver, and someone should do it with UV-B5. They may very well be the first drivers with NEEDS_COMPAT_SERIAL flipped to false :)
Thanks, Zhaofeng Li
On Fri, Feb 8, 2019 at 7:08 PM Dan Smith via chirp_devel chirp_devel@intrepid.danplanet.com wrote:
I still have most of the code for PyGObject. All of the UI works (tested quite thoroughly), but populating the list is a LOT slower, which I've been stuck on for a while and wanted to fix before putting it up. I'll clean it up a bit and drop the current version somewhere tonight, but if we want to use it, it'll take some effort to merge it with current work.
That might be interesting to look at. There is also a giant monolithic patch in #6363 that has some pygi changes, but it pretty much doesn't work at all for me and seems like it might just be the result of running multiple autoconversion tools on the tree.
What I have in the py3 branch does operate against PyGObject, using the pygtk compatibility layer. It's far from perfect, but it's close enough that I feel like it's a good option for the time being.
If we get to the point of being able to run under py3 as well as py2, then converting to all native PyGObject would be good of course.
--Dan _______________________________________________ chirp_devel mailing list chirp_devel@intrepid.danplanet.com http://intrepid.danplanet.com/mailman/listinfo/chirp_devel Developer docs: http://chirp.danplanet.com/projects/chirp/wiki/Developers
After some struggle, I managed to port the Baofeng UV-B5 and TYT TH-350 (subclass of UV-B5) drivers to Python 3, along with various other fixes to the codebase. Attached is the patch.
Thanks, I skimmed it and see a couple of things I have already fixed in my local tree but haven't pushed. So I may not apply those from your patch, depending.
I've tested uploading, downloading and saving with the TH-350 driver, and someone should do it with UV-B5. They may very well be the first drivers with NEEDS_COMPAT_SERIAL flipped to false :)
Ah, dangit, you stole the crown!
But seriously, thanks for doing that. I will look in more detail a little later today!
--Dan
After some struggle, I managed to port the Baofeng UV-B5 and TYT TH-350 (subclass of UV-B5) drivers to Python 3, along with various other fixes to the codebase. Attached is the patch.
Comments below:
# HG changeset patch # User Zhaofeng Li hello@zhaofeng.li # Date 1549691138 28800 # Fri Feb 08 21:45:38 2019 -0800 # Branch py3 # Node ID 838216796b5a7e014674af9b16751cec87fb73e9 # Parent 7d5cf3c3432794c5ea4b1d1d2cc65d6038d3a73b [py3] bitwise.py: Python 3-ize
Note that I've been using issue #495 to track all pure python3 conversions, as that is the earliest related issue I could find. I'll add it to your patches.
Also, this patch breaks the settings tests on uvb5 on python2, and I want to try to keep all of those working as much as possible as that is one of the indicators of when we'll be safe to merge with the main branch.
diff -r 7d5cf3c34327 chirp/bitwise.py --- a/chirp/bitwise.py Fri Feb 08 20:30:10 2019 -0800 +++ b/chirp/bitwise.py Sat Feb 09 08:18:14 2019 -0800 @@ -232,51 +232,51 @@
class arrayDataElement(DataElement):
<snip>
def get_raw(self):
return "".join([item.get_raw() for item in self.__items])
return b"".join([item.get_raw() for item in self.__items])
I don't think we can make this change because everywhere else that depends on this returning strings will break. Obviously there are other things in bitwise that need to get cleaned up like this as well.
How about we make the function take a bytes=False, which will control what it returns? It's not ideal, but we can put a comment there and use it later to track down things still calling it looking for strings. I think we also need to make sure that whatever it is iterating also returns bytes, because in the case of a string array this will try to join bytes and str. I can take a look at fixing that.
It would be nicer to be able to control bitwise's return values as a whole at parse-time, but that whole thing is a bit of a mess and I don't really want to mess with it.
# HG changeset patch # User Zhaofeng Li hello@zhaofeng.li # Date 1549691367 28800 # Fri Feb 08 21:49:27 2019 -0800 # Branch py3 # Node ID 3434f35e76cfc7dd481a46cc61b9c9c999d01e0f # Parent aa4dd87eb5d0640e715aa2e70387161138b26885 [py3] uvb5.py: 2to3 with manual fixes
<snip>
def get_memory(self, number): _mem, _nam = self._get_memobjs(number) mem = chirp_common.Memory() if isinstance(number, str): mem.number = self.SPECIALS[number] mem.extd_number = number else: mem.number = number
if _mem.freq.get_raw()[0] == "\xFF":
if _mem.freq.get_raw()[0] == 0xff:
This won't work on python2:
$ python -c 'print(b"\x00"[0] == 0)' False $ python3 -c 'print(b"\x00"[0] == 0)' True
This is one of the super frustrating parts of the conversion, because while 2.7 is supposed to be compatible with 3, these kinds of things will quietly introduce bugs :(
# HG changeset patch # User Zhaofeng Li hello@zhaofeng.li # Date 1549694726 28800 # Fri Feb 08 22:45:26 2019 -0800 # Branch py3 # Node ID d3ba01e9b6ef486bc7b66bb2d07602ff65f48124 # Parent 8d77855fe3ecb0503388d628014eac9a274d7822 [py3] th350.py: 2to3, with many manual fixes
This breaks a couple of tests on on the th350 driver for python2.
Okay, so I'm going to apply these:
[py3] chirp_common.py: Fix file saving [py3] memmap.py: Fix slicing [py3] memedit.py: Convert power to str
and hold off on:
[py3] bitwise.py: Python 3-ize [py3] uvb5.py: 2to3 with manual fixes [py3] uvb5.py: Adapt for new Serial [py3] th350.py: 2to3, with many manual fixes
for the moment. I haven't looked at what the problem is with the first bitwise patch, but I will, along with working on the get_raw(bytes=true) stuff.
Thanks!
--Dan
# HG changeset patch # User Zhaofeng Li hello@zhaofeng.li # Date 1549691138 28800 # Fri Feb 08 21:45:38 2019 -0800 # Branch py3 # Node ID 838216796b5a7e014674af9b16751cec87fb73e9 # Parent 7d5cf3c3432794c5ea4b1d1d2cc65d6038d3a73b [py3] bitwise.py: Python 3-ize
Note that I've been using issue #495 to track all pure python3 conversions, as that is the earliest related issue I could find. I'll add it to your patches.
Also, this patch breaks the settings tests on uvb5 on python2, and I want to try to keep all of those working as much as possible as that is one of the indicators of when we'll be safe to merge with the main branch.
Ah, it's because a lot of places still need the __div__ you removed, especially py2. I'll add it back in and make it work like __floordiv__.
--Dan
I've fixed the drivers to work on both Python 2 and 3, with all tests passing. Attached is the patch.
To ensure that the behavior of get_raw(asbytes=True) is consistent across Python 2 and 3, I've pulled in python-future which provides a mostly-complete implementation of bytes for Python 2. This way we can guarantee that bytes or bytes-compatible objects are returned when new-style drivers ask for them.
Another thing that I missed is importing __futures__.division for Python 2, again to ensure consistent behavior in 2/3.
Zhaofeng Li
On Sat, Feb 9, 2019 at 10:38 AM Dan Smith via chirp_devel chirp_devel@intrepid.danplanet.com wrote:
# HG changeset patch # User Zhaofeng Li hello@zhaofeng.li # Date 1549691138 28800 # Fri Feb 08 21:45:38 2019 -0800 # Branch py3 # Node ID 838216796b5a7e014674af9b16751cec87fb73e9 # Parent 7d5cf3c3432794c5ea4b1d1d2cc65d6038d3a73b [py3] bitwise.py: Python 3-ize
Note that I've been using issue #495 to track all pure python3 conversions, as that is the earliest related issue I could find. I'll add it to your patches.
Also, this patch breaks the settings tests on uvb5 on python2, and I want to try to keep all of those working as much as possible as that is one of the indicators of when we'll be safe to merge with the main branch.
Ah, it's because a lot of places still need the __div__ you removed, especially py2. I'll add it back in and make it work like __floordiv__.
--Dan _______________________________________________ chirp_devel mailing list chirp_devel@intrepid.danplanet.com http://intrepid.danplanet.com/mailman/listinfo/chirp_devel Developer docs: http://chirp.danplanet.com/projects/chirp/wiki/Developers
I've fixed the drivers to work on both Python 2 and 3, with all tests passing. Attached is the patch.
Sweet, thanks
diff -r 6147365892e1 -r 6f5c2af0b6f1 chirp/drivers/uvb5.py --- a/chirp/drivers/uvb5.py Fri Feb 08 21:49:27 2019 -0800 +++ b/chirp/drivers/uvb5.py Fri Feb 08 22:42:54 2019 -0800
<snip>
- data = radio._mmap[0x0030:]
- data = radio._mmap.get_byte_compatible()[0x0030:]
Not that it matters, but this shouldn't be strictly necessary since you're using MemoryMapBytes right?
diff -r 6f5c2af0b6f1 -r d1470be72a27 chirp/drivers/th350.py --- a/chirp/drivers/th350.py Fri Feb 08 22:42:54 2019 -0800 +++ b/chirp/drivers/th350.py Sat Feb 09 17:49:11 2019 -0800
<snip>
def do_download(radio): do_ident(radio)
- data = "TH350 Radio Program data v1.08\x00\x00"
- data += ("\x00" * 16)
- data = b"TRI350 Radio Program data v1.08\x00"
- data += (b"\x00" * 16)
Why did you change this and the match_model stuff? In the past, we definitely couldn't do something like this because it would make images created by newer chirp incompatible with older chirp for no real reason. Since your driver only existed after the metadata change, this doesn't really affect you, but I don't really want to get into the habit of doing this without good reason.
To ensure that the behavior of get_raw(asbytes=True) is consistent across Python 2 and 3, I've pulled in python-future which provides a mostly-complete implementation of bytes for Python 2. This way we can guarantee that bytes or bytes-compatible objects are returned when new-style drivers ask for them.
This is cool, I spent some time making sure this is installable and functional in the win32 py2 environment and it appears to be, so this should help a lot.
Another thing that I missed is importing __futures__.division for Python 2, again to ensure consistent behavior in 2/3.
I was going to take exception to this, but TIL that the __futures__ imports like that only affect the module in which they were imported, so this too is a good thing.
I pushed all but the th350 patch, just looking for information on the format change.
Thanks!
--Dan
- data = radio._mmap[0x0030:]
- data = radio._mmap.get_byte_compatible()[0x0030:]
Not that it matters, but this shouldn't be strictly necessary since you're using MemoryMapBytes right?
This is necessary because the driver is still given the old MemoryMap when the user loads the image from a file, and will break uploading on Python 3. Ideally, the directory module should initialize the driver with MemoryMapBytes based on a class attribute.
On another note, MemoryMapBytes returns str on Python 2, and we should probably make it return python-future bytes. Many breakages may occur so this should be tested thoroughly.
def do_download(radio): do_ident(radio)
- data = "TH350 Radio Program data v1.08\x00\x00"
- data += ("\x00" * 16)
- data = b"TRI350 Radio Program data v1.08\x00"
- data += (b"\x00" * 16)
Why did you change this and the match_model stuff? In the past, we definitely couldn't do something like this because it would make images created by newer chirp incompatible with older chirp for no real reason. Since your driver only existed after the metadata change, this doesn't really affect you, but I don't really want to get into the habit of doing this without good reason.
This is the header that the official Windows programmer uses, and the change will allow CHIRP images to be read by the official app (and vice versa). I really should have noticed this earlier when I first submitted the driver.
Thanks, Zhaofeng Li
This is necessary because the driver is still given the old MemoryMap when the user loads the image from a file, and will break uploading on Python 3. Ideally, the directory module should initialize the driver with MemoryMapBytes based on a class attribute.
Ah, okay.
On another note, MemoryMapBytes returns str on Python 2, and we should probably make it return python-future bytes. Many breakages may occur so this should be tested thoroughly.
Yep, I think I'm going to need to do that for the icom driver fixes I'm doing right now, so I'll put this in the queue.
This is the header that the official Windows programmer uses, and the change will allow CHIRP images to be read by the official app (and vice versa). I really should have noticed this earlier when I first submitted the driver.
Okay, good reason. I'll be pushing that last up in a bit.
--Dan
participants (4)
-
Dan Clemmensen -
Dan Smith -
Jake Merdich -
Zhaofeng Li