Hello all, TIA for any help you can give me below ;)
Firstly, great software, really useful, Thanks! Secondly, I'm not an accomplished python programmer, but have spend a while doing embedded C on micro's. So most likely I'm going to ask a naive question. Third, I'm attempting to get a new driver for an ICOM M710, it's a commercial HF Marine SSB radio that has limited support, but there are a few of us still using them.
That said, I've got part of the driver stood up, I can query it over the UART and get the memory dump. I've done a fair amount of sleuthing through the data and have a mild handle on the formatting of the data blob that comes out during download. I get a little less that 10,000 8 bit bytes from the rig, it's almost like a memory dump.
Is the intent of MemoryMapBytes to hold an "image" of that dump, but provide some structure to the blob for easier access?
It appears there are two channels per "record", there seems to be a record start marker with some channel information, 2x rx frequency, mode, name, and tx frequency then a checksum and end of record marker.
I've setup my memory format like this: MEM_FORMAT = """ struct mem_struct{ char rx_freq[9]; char mode; char name[14]; char tx_freq[8]; };
seekto 0x0000; struct { char unknown_SOR[11]; struct mem_struct channels[2]; char checksum[2]; char unknown_EOR; }memory[2]; """
So, to detail my question a bit more. I have a byte array that holds the incoming image that might look something like this: annotated with comments
#Channel 0 RX 12345.6 TX 27654.3 USB ABCDEFG FE FE EF EE E4 45 30 32 30 32 30 #?? start of record?? 31 32 33 34 35 36 30 30 30 #12345.6 30 #USB = 30, R3E = 31, AM = 32, LSB = 33, AFS = 34, FSK = 35, CW = 36 34 31 34 32 34 33 34 34 34 35 34 36 34 37 #ABCDEFG 32 37 36 35 34 33 30 30 #27654.3
#channel 1 RX 11111,1 TX 22222.2 31 31 31 31 31 31 30 30 30 #11111.1 30 #USB = 30 LSB = 33 34 38 34 39 34 41 34 42 34 43 34 44 34 45 #HIJKLMN 32 32 32 32 32 32 30 30 #22222.2 46 30 #?? this chaned when the mode changed checksum? FD #?? end of record ???
def do_download(radio): """This is your download function"""
#this is working, but no reason to keep hammering the radio right now #todo uncomment this later #init_download = bytes([0xFE, 0xFE, 0xEE, 0xEF, 0xE2, 0x16, 0x32, 0x00, 0x01, 0xFD]) #send(radio, init_download) #ack = radio.pipe.read(10) #check the ack
#todo read until FDFEFEEFEEE5FD is received. #data = bytes() #for _i in range(0, 9838): # data += radio.pipe.read(1)
# this is just what part of what the incoming data looks like, this is record 1 memory_image = ( [0xFE,0xFE,0xEF,0xEE,0xE4,0x45,0x30,0x32,0x30,0x32,0x30,0x31,0x32,0x33,0x34,0x35, 0x36,0x30,0x30,0x30,0x30,0x34,0x31,0x34,0x32,0x34,0x33,0x34,0x34,0x34,0x35,0x34, 0x36,0x34,0x37,0x32,0x37,0x36,0x35,0x34,0x33,0x30,0x30,0x31,0x31,0x31,0x31,0x31, 0x31,0x30,0x30,0x30,0x30,0x34,0x38,0x34,0x39,0x34,0x41,0x34,0x42,0x34,0x43,0x34, 0x44,0x34,0x45,0x32,0x32,0x32,0x32,0x32,0x33,0x30,0x30,0x46,0x32,0xFD])
print(memory_image) data = b"" data = memory_image
print(data) #return data return memmap.MemoryMapBytes(data)
//////////////////////////// output //////////////////////////////
[254, 254, 239, 238, 228, 69, 48, 50, 48, 50, 48, 49, 50, 51, 52, 53, 54, 48, 48, 48, 48, 52, 49, 52, 50, 52, 51, 52, 52, 52, 53, 52, 54, 52, 55, 50, 55, 54, 53, 52, 51, 48, 48, 49, 49, 49, 49, 49, 49, 48, 48, 48, 48, 52, 56, 52, 57, 52, 65, 52, 66, 52, 67, 52, 68, 52, 69, 50, 50, 50, 50, 50, 51, 48, 48, 70, 50, 253] [254, 254, 239, 238, 228, 69, 48, 50, 48, 50, 48, 49, 50, 51, 52, 53, 54, 48, 48, 48, 48, 52, 49, 52, 50, 52, 51, 52, 52, 52, 53, 52, 54, 52, 55, 50, 55, 54, 53, 52, 51, 48, 48, 49, 49, 49, 49, 49, 49, 48, 48, 48, 48, 52, 56, 52, 57, 52, 65, 52, 66, 52, 67, 52, 68, 52, 69, 50, 50, 50, 50, 50, 51, 48, 48, 70, 50, 253] ERROR: Failed to clone: Traceback (most recent call last): File "G:\Projects\HamRadio\Repo\chirp\chirp\wxui\clone.py", line 93, in run self._fn() File "G:\Projects\HamRadio\Repo\chirp\chirp\drivers\icom_m710.py", line 202, in sync_in self._mmap = do_download(self) ^^^^^^^^^^^^^^^^^ File "G:\Projects\HamRadio\Repo\chirp\chirp\drivers\icom_m710.py", line 121, in do_download return memmap.MemoryMapBytes(data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "G:\Projects\HamRadio\Repo\chirp\chirp\memmap.py", line 26, in __init__ assert isinstance(data, bytes) ^^^^^^^^^^^^^^^^^^^^^^^ AssertionError WARNING: Stopping clone thread
I get a little less that 10,000 8 bit bytes from the rig, it's almost like a memory dump.
Almost all of the radios CHIRP supports are dumping memory images, so.. yes :)
Is the intent of MemoryMapBytes to hold an "image" of that dump, but provide some structure to the blob for easier access?
Yes, but technically MemoryMapBytes is just the flag image, the output of bitwise.parse() is the logical structure object laid on top of the flat memory buffer. In C parlance, the MemoryMapBytes is the result of the malloc() and the memobj (output of bitwise.parse()) is like a `struct mymemobj*` you point at it to get a structured view.
It appears there are two channels per "record", there seems to be a record start marker with some channel information, 2x rx frequency, mode, name, and tx frequency then a checksum and end of record marker.
I've setup my memory format like this: MEM_FORMAT = """ struct mem_struct{ char rx_freq[9]; char mode; char name[14]; char tx_freq[8]; };
We would call this one channel. Lots of radios store the tx and rx frequency for a given channel (i.e. a numbered memory channel in the radio) separately.
seekto 0x0000; struct { char unknown_SOR[11]; struct mem_struct channels[2]; char checksum[2]; char unknown_EOR; }memory[2]; """
Er, I don't think it makes much sense to have a mem_struct[2] in the middle of this. I'm guessing maybe you're confusing the packetization of the memory data over the wire with the actual memory structure? On all (amateur, granted) Icom radios, they use the ICF protocol to transfer the memory contents to the PC in 8, 16 or 32-byte chunks (depending on the vintage of the radio). Each chunk has a sort of SOR, src and dest addresses (since it's sort of a networking protocol), a command number, the actual payload, and an EOR. You don't want to store all of that, you want to just store the payload in the proper index in the MemoryMapBytes. Other radios with similar protocols include a command, address (i.e. where in the memory address space this chunk goes), length, checksum, etc.
Have a look at icf.py, specifically IcfFrame.parse() to see if it happens to match what you're looking at. At the very least, you should implement that sort of scheme, where you speak the protocol separately from the actual payload that you store.
--Dan
Yes, but technically MemoryMapBytes is just the flag image, the output of bitwise.parse() is the logical structure object laid on top of the flat memory buffer. In C parlance, the MemoryMapBytes is the result of the malloc() and the memobj (output of bitwise.parse()) is like a `struct mymemobj*` you point at it to get a structured view.
Great, that's what I was thinking it was.
Er, I don't think it makes much sense to have a mem_struct[2] in the middle of this. I'm guessing maybe you're confusing the packetization of the memory data over the wire with the actual memory structure? On all (amateur, granted) Icom radios, they use the ICF protocol to transfer the memory contents to the PC in 8, 16 or 32-byte chunks (depending on the vintage of the radio). Each chunk has a sort of SOR, src and dest addresses (since it's sort of a networking protocol), a command number, the actual payload, and an EOR. You don't want to store all of that, you want to just store the payload in the proper index in the MemoryMapBytes. Other radios with similar protocols include a command, address (i.e. where in the memory address space this chunk goes), length, checksum, etc.
Have a look at icf.py, specifically IcfFrame.parse() to see if it happens to match what you're looking at. At the very least, you should implement that sort of scheme, where you speak the protocol separately from the actual payload that you store.
I'll take a look at the icf file, make there are clues in there. The core of my question was answered above, I just have to figure out how to get the overlay to work with the data I have in whatever structure it ends up in.
I think there are two protocols remote and clone, remote looks like NEMA ICF for remote control, there doesn't appear to be a way to set memories using the remote protocol. the clone protocol is the one the old software uses and it just sends a single to dump the entire memory image. From what it looks like sniffing the UART, the same 10K of bytes go back down The software for the M710 is used for cloning and you send one command and 10K bytes of data come back. When you program it, it sends a similar command and all 10K bytes go back down. i.e. you send FE, FE, EE, EF, E2, 16, 32, 00, 01, FD and get 10K bytes back then you send FE FE EE EF E3 16 32 00 01 FD + 10K bytes to program the image. What we have now is fairly painful, we have to install dosbox and run an ancient dos program to program the memories.
Yes, but technically MemoryMapBytes is just the flag image, the output of bitwise.parse() is the logical structure object laid on top of the flat memory buffer. In C parlance, the MemoryMapBytes is the result of the malloc() and the memobj (output of bitwise.parse()) is like a `struct mymemobj*` you point at it to get a structured view.
Great, that's what I was thinking it was.
Note I meant "flat image" not "fag image" above.
I think there are two protocols remote and clone, remote looks like NEMA ICF for remote control, there doesn't appear to be a way to set memories using the remote protocol.
Yep, it's common to have both on radios like this.
the clone protocol is the one the old software uses and it just sends a single to dump the entire memory image. From what it looks like sniffing the UART, the same 10K of bytes go back down The software for the M710 is used for cloning and you send one command and 10K bytes of data come back. When you program it, it sends a similar command and all 10K bytes go back down.
Yep, this is how the ICF clone radios work. You send a (properly framed) 0xE2 (clone out) command and then it spews the memory format to you in chunks framed in 0xE4 commands (clone data).
i.e. you send FE, FE, EE, EF, E2, 16, 32, 00, 01, FD and get 10K bytes back then you send FE FE EE EF E3 16 32 00 01 FD + 10K bytes to program the image. What we have now is fairly painful, we have to install dosbox and run an ancient dos program to program the memories.
Yep, this looks exactly like the ICF as implemented by those drivers. Basically every icom clone-type radio in the tree is handled by a single protocol handler in icf.py. If yours behaves the same, you can likely use it to get the entire memory image cleaned of all the framing, and use it to program it back. Please do not try to reimplement it in your driver, and definitely don't store the framing (i.e. SOR, addresses, command, EOR) bits in the memory image.
Also just FYI, the Icom HF radios in the tree don't use this, they get/set the memories over CIV (i.e the remote control protocol) so I would look at some of the regular drivers (i.e ic2820.py) for example. You'll notice they include no upload/download code because it's all handled by the base class (IcomCloneModeRadio).
--Dan
I knew what you meant! Yes, i would prefer not to implement the entire ICF protocol again, thanks, but no thanks. I already have to do the entire remote protocol for another effort.
I originally went looking at CAT, but it was clear that wasn't the way to go, the icf looks very promising, I already have the "radio: I can't talk to this model" ;)
Thanks for pointing me to the ICF, I think that should work a lot better .
Paul
I originally went looking at CAT, but it was clear that wasn't the way to go, the icf looks very promising, I already have the "radio: I can't talk to this model" ;)
Yep, set the _model bytes on the class and then it should get farther. You'll also need _memsize to get a full image, and _ranges to do an upload. Some of the older radios require holes in the uploaded image to successfully upload.
--Dan
I've made some progress on the m710 driver today. I'm able to read a good portion of the memories and get them to the display, Like you said, there may be some memory holes I need to deal with. I posted a picture of the read here:
https://drive.google.com/file/d/1lwDVUii8_hCyOopGKgZubncMHW1Y7WyU/view?usp=s...
Is there a way to remove the Tone, Duplex, and Skip columns and then add a TX Frequency column?
Thanks, Paul
Is there a way to remove the Tone, Duplex, and Skip columns and then add a TX Frequency column?
This is driven from the feature flags in the RadioFeatures object you return from get_features(), although Tone and Duplex are very universal, so the UI may not actually hide those columns if you fully don't support them. I'll have a look. CHIRP supports a TX frequency via duplex=split, offset=$tx_freq because it was really designed for typical VHF/UHF ham radios with a duplex/offset sort of arrangement. My recommendation for best compatibility would be that you support simplex and duplex=split. In the former, you internally set tx_freq=freq and if the user selects the latter, then you take tx_freq from the offset field. Users familiar with CHIRP will know that's how they select a TX frequency, and selecting duplex=split will prompt them for such.
So:
rf.valid_tmodes = [] rf.valid_skips = [] rf.valid_duplexes = ['', 'split']
I'll have a look at the UI and will probably need to make it hide the tone column in your case.
--Dan
skips is gone now, tones still shows up, but not a big deal. Duplex is where I was going as well, I noticed a little popup that sets the TX frequency. When it is in simplex mode, it looks like the TX frequency is set to FFFFFF, so that can be an indicator of duplex/simplex.
Tomorrow I can see if i can program the radio without bricking it! Maybe I'll send the serial bytes to a file and just validate they look good before sending them on to the radio.
Thanks for the guidance. Paul
Dan, I was able to download to the radio. I need to look into it more, but icf.py line 906 self.pipe.baudrate = 9600, seemed to trip up the download. The M710 uses 4800 baud also what the icom clone software uses. Once I changed that to 4800, I was able to write out to the radio and my changes were sticky!
There are still a number of things to clean up, There are radio settings to work out as well, and some advanced channels I need to figure out, but, not bad!
I see there is actually an issue for this radio https://chirpmyradio.com/issues/1067 Should I make a branch or wait until I have full functionality and submit?
Paul
I was able to download to the radio. I need to look into it more, but icf.py line 906 self.pipe.baudrate = 9600, seemed to trip up the download. The M710 uses 4800 baud also what the icom clone software uses. Once I changed that to 4800, I was able to write out to the radio and my changes were sticky!
Cool, congrats.
I would just hack around this by overriding sync_out() in your driver:
def sync_out(self): radio.pipe.baudrate = 4800 icf.clone_to_radio(self)
Technically if you have BAUD_RATE set on your class this should be done for you, but since this is a divergence from every other icom radio in the tree, it's probably good to just be explicit here.
There are still a number of things to clean up, There are radio settings to work out as well, and some advanced channels I need to figure out, but, not bad!
I see there is actually an issue for this radio https://chirpmyradio.com/issues/1067 Should I make a branch or wait until I have full functionality and submit?
I would very much prefer an initial driver with just basic functionality and incremental changes to add more iteratively.
--Dan
Hi Dan, Diving a little deeper into the M710. The radio takes the clone. Stepping through icf.py, line 492, every frame goes down to the radio, when the CMD_CLONE_END is sent, the display on the radio says: "Good CLONE". The radio responds with an echo of the CLONE_END command and that's it. I did a port capture of the old software and it shows the same CLONE_END command and the same response back.
Is there a way that I can rework that area after icf.py line 492 so that I don't get an error dialog at the end of a clone? I did work around it by adding an _ignore_clone_ok to class IcomCloneModeRadio, then just returning after reading the final frame.
Paul
Diving a little deeper into the M710. The radio takes the clone. Stepping through icf.py, line 492, every frame goes down to the radio, when the CMD_CLONE_END is sent, the display on the radio says: "Good CLONE". The radio responds with an echo of the CLONE_END command and that's it. I did a port capture of the old software and it shows the same CLONE_END command and the same response back.
Is there a way that I can rework that area after icf.py line 492 so that I don't get an error dialog at the end of a clone? I did work around it by adding an _ignore_clone_ok to class IcomCloneModeRadio, then just returning after reading the final frame.
It'd help to see code and the actual trace/log of what's going on. Are you saying the M710 isn't sending the CMD_CLONE_OK frame? If so, a flag to avoid waiting for it seems reasonable, yeah.
--Dan
That's correct, after sending the CMD_CLONE_END (0xE5), the display of the radio says "Good CLONE" and nothing else comes out of the radio. I verified the same behavior with the Icom programming software, after all the frames, the clone end (0xE5) is sent and the radio doesn't send anything back.
I put together a draft pull request. I'm not sure how to run the unit tests, if you can point me in the right direction, I'll be happy to run them.
participants (2)
-
Dan Smith -
kg7hf1＠gmail.com