[chirp_devel] [PATCH] [FT-70] New Model #5329 Increased second block size
# HG changeset patch # User Nicolas Pike nicolas.jon.pike@gmail.com # Date 1521501092 0 # Mon Mar 19 23:11:32 2018 +0000 # Node ID c290bb32d9bf8262b20e43c797dd71f8fe363c73 # Parent 61ba9c8151706bad0a270ee94acd3f16417854e3 [FT-70] New Model #5329 Increased second block size Block size not large enough for some Wndows installs Temporary fix, investigating further.
diff -r 61ba9c815170 -r c290bb32d9bf chirp/drivers/ft70.py --- a/chirp/drivers/ft70.py Thu Mar 15 23:41:07 2018 +0000 +++ b/chirp/drivers/ft70.py Mon Mar 19 23:11:32 2018 +0000 @@ -466,14 +466,14 @@ @directory.register class FT70Radio(yaesu_clone.YaesuCloneModeRadio): """Yaesu FT-70DE""" - BAUD_RATE = 115200 + BAUD_RATE = 38400 VENDOR = "Yaesu" MODEL = "FT-70D"
_model = "AH51G"
_memsize = 65227 # 65227 read from dump ? - _block_lengths = [10, 65555] # ????? Not sure why this works to match _memsize + _block_lengths = [10, 65920] # ????? Not sure why this works to match _memsize _block_size = 32 _mem_params = (900, # size of memories array 900, # size of flags array
diff -r 61ba9c815170 -r c290bb32d9bf chirp/drivers/ft70.py --- a/chirp/drivers/ft70.py Thu Mar 15 23:41:07 2018 +0000 +++ b/chirp/drivers/ft70.py Mon Mar 19 23:11:32 2018 +0000 @@ -466,14 +466,14 @@ @directory.register class FT70Radio(yaesu_clone.YaesuCloneModeRadio): """Yaesu FT-70DE"""
- BAUD_RATE = 115200
- BAUD_RATE = 38400
Does the radio really magically operate at either of these two rates? Or do you set it in the radio before you do the clone? Otherwise, I can't think of why this could have been set to the old value, or be set to this new value without breaking things. We have a couple of radios that can be at one of several rates and we probe until we determine the proper one. Just launching into a clone at whatever rate you pick here seems unlikely to work. Can you explain more?
VENDOR = "Yaesu" MODEL = "FT-70D" _model = "AH51G" _memsize = 65227 # 65227 read from dump ?
- _block_lengths = [10, 65555] # ????? Not sure why this works to match _memsize
- _block_lengths = [10, 65920] # ????? Not sure why this works to match _memsize
This writes less data to the radio, that's it. It should be very very not-operating-system specific. If you need to write the same amount of data in smaller chunks, you need to set _block_size to something else (although it's really small by default already). However, this looks really suspicious to me...
--Dan
Hi,
On 21 March 2018 at 22:07, Dan Smith via chirp_devel <chirp_devel@intrepid. danplanet.com> wrote:
diff -r 61ba9c815170 -r c290bb32d9bf chirp/drivers/ft70.py --- a/chirp/drivers/ft70.py Thu Mar 15 23:41:07 2018 +0000 +++ b/chirp/drivers/ft70.py Mon Mar 19 23:11:32 2018 +0000 @@ -466,14 +466,14 @@ @directory.register class FT70Radio(yaesu_clone.YaesuCloneModeRadio): """Yaesu FT-70DE"""
- BAUD_RATE = 115200
- BAUD_RATE = 38400
Does the radio really magically operate at either of these two rates? Or do you set it in the radio before you do the clone? Otherwise, I can't think of why this could have been set to the old value, or be set to this new value without breaking things. We have a couple of radios that can be at one of several rates and we probe until we determine the proper one. Just launching into a clone at whatever rate you pick here seems unlikely to work. Can you explain more?
The FT70 has a built in USB interface The Chirp driver is only intended to be used with this, no baud rate is set on it. Using a higher baud rate made a slight difference in speed, But initially thought this might now be causing other issues, so changed back to 38400.
VENDOR = "Yaesu" MODEL = "FT-70D" _model = "AH51G" _memsize = 65227 # 65227 read from dump ?
- _block_lengths = [10, 65555] # ????? Not sure why this works to
match _memsize
- _block_lengths = [10, 65920] # ????? Not sure why this works to
match _memsize
This writes less data to the radio, that's it. It should be very very not-operating-system specific. If you need to write the same amount of data in smaller chunks, you need to set _block_size to something else (although it's really small by default already). However, this looks really suspicious to me...
*At 38400 or 15200 baud blocks were being returned with no data in them.*
Investigated further with Fred and Mark
We are losing blocks at the beginning in the Chirp yaesu_clone method.
[2018-03-19 21:35:41,483] chirp.drivers.yaesu_clone - DEBUG: Read 0/65920 [2018-03-19 21:35:41,733] chirp.drivers.yaesu_clone - DEBUG: Read 0/65920 [2018-03-19 21:35:41,986] chirp.drivers.yaesu_clone - DEBUG: Read 0/65920 [2018-03-19 21:35:42,236] chirp.drivers.yaesu_clone - DEBUG: Read 0/65920
I thought the rest of the download was working, hence the puzzle of how are we short at the end. Which did not make sense if we were only losing blocks at the beginning...
After increasing the size last night it worked and then we realised that blocks must be being returned with no data during the rest of the download. Slight confusion of the debug info as that shows the running total - not the bytes read for each block - errors will show as an unchanged value not as zero... Loaded it into a spreadsheet this morning and behold we are losing a dozen or so blocks...
Two examples
[2018-03-19 21:18:16,013] chirp.drivers.yaesu_clone - DEBUG: Read 49152/65920 [2018-03-19 21:18:16,263] chirp.drivers.yaesu_clone - DEBUG: Read 49152/65920
[2018-03-19 21:18:16,489] chirp.drivers.yaesu_clone - DEBUG: Read 53248/65920 [2018-03-19 21:18:16,739] chirp.drivers.yaesu_clone - DEBUG: Read 53248/65920
etc. Note the read figure has not changed, so as you said that block read did not return any data.
And guess what they are on 4k boundaries.....
Slightly different behaviour on Mac vs Windows but only how many blocks get lost... Which as Fred says gets us to how the O/S or radio are handling the requests.
I submitted a patch to increase the size - for the time being - but it seems that yaesu_clone needs looking at. This doubtless explains the odd number in the FT1 driver which the FT-70 driver is "very" loosely based on - I never did understand how that number was arrived at.....
Fred has now done some great work investigating this further, for a full fix, as opposed to the temp of just reading more blocks.
*Freds observations*
Think I found what the problem is with read. chirp sets the time out to .25 secs I changed it to 2 secs and seems to work.
I did not make a patch, there seems to be numerous changes with hg diff anyway from context you will see where the changes are.
_model = "AH51G"
_memsize = 65227 # 65227 read from dump ? _block_lengths = [10, 65217] # ????? Not sure why this works to match _memsize _block_size = 32 _mem_params = (900, # size of memories array 900, # size of flags array )
_has_vibrate = False _has_af_dual = True
# not sure of correct place to set timeout def sync_in(self): self.pipe.timeout=2 super(FT70Radio, self).sync_in()
def process_mmap(self):
mem_format = MEM_SETTINGS_FORMAT + MEM_FORMAT + MEM_CALLSIGN_FORMAT + MEM_CHECKSUM_FORMAT
self._memobj = bitwise.parse(mem_format % self._mem_params, self._mmap)
We are going to test this further and submit another patch. I submitted the previous patch as a temp measure, as it was broken on many - (but not my!!) Windows installs
Phew!
Nicolas
The FT70 has a built in USB interface The Chirp driver is only intended to be used with this, no baud rate is set on it. Using a higher baud rate made a slight difference in speed, But initially thought this might now be causing other issues, so changed back to 38400.
That is very surprising to me. Every other USB serial device I know of honors the baud rate as expected. Unless the radio is really mirroring whatever it can tell the USB serial bridge is set to, but that would be the first such device I've ever encountered.
At 38400 or 15200 baud blocks were being returned with no data in them.
So the problem is with _reading_ from the radio, not writing to it? In almost all cases of flaky "it works on my system but not his" cases, it was writing to the radio that was the problem. This is usually because the radio has subtle timing requirements (or a small buffer with no flow control). So, odd thing #2.
Investigated further with Fred and Mark We are losing blocks at the beginning in the Chirp yaesu_clone method.
[2018-03-19 21:35:41,483] chirp.drivers.yaesu_clone - DEBUG: Read 0/65920 [2018-03-19 21:35:41,733] chirp.drivers.yaesu_clone - DEBUG: Read 0/65920 [2018-03-19 21:35:41,986] chirp.drivers.yaesu_clone - DEBUG: Read 0/65920 [2018-03-19 21:35:42,236] chirp.drivers.yaesu_clone - DEBUG: Read 0/65920
I thought the rest of the download was working, hence the puzzle of how are we short at the end. Which did not make sense if we were only losing blocks at the beginning...
These aren't blocks, they're chunks, and the yaesu_clone code assumes that each chunk of the whole block can and will be read. Sounds like that code should at _least_ assert that it read the length expected, because if it doesn't, it will silently return less data than the block requested. In reality, it should probably be smarter and plan to read the total number of bytes regardless of how many trips it takes, and have a overall timeout to catch the case where we don't hear anything from the radio.
After increasing the size last night it worked and then we realised that blocks must be being returned with no data during the rest of the download. Slight confusion of the debug info as that shows the running total - not the bytes read for each block - errors will show as an unchanged value not as zero... Loaded it into a spreadsheet this morning and behold we are losing a dozen or so blocks...
Two examples
[2018-03-19 21:18:16,013] chirp.drivers.yaesu_clone - DEBUG: Read 49152/65920 [2018-03-19 21:18:16,263] chirp.drivers.yaesu_clone - DEBUG: Read 49152/65920
[2018-03-19 21:18:16,489] chirp.drivers.yaesu_clone - DEBUG: Read 53248/65920 [2018-03-19 21:18:16,739] chirp.drivers.yaesu_clone - DEBUG: Read 53248/65920
etc. Note the read figure has not changed, so as you said that block read did not return any data.
Yeah, so what I'm saying about changing the behavior of _chunk_read() above would apply I think.
I submitted a patch to increase the size - for the time being - but it seems that yaesu_clone needs looking at.
Okay, so I had this backwards because I thought you were talking about uploading not downloading. So I understand why that being larger makes a difference, but it'd be wrong for anyone that doesn't stall at the beginning. I guess the same stupidness in _chunk_read() makes that not completely hang for those people, but...not the right solution :)
chirp sets the time out to .25 secs I changed it to 2 secs and seems to work.
Yes, and some drivers have to change this because of their radios (although it's always on write AFAIK). You could do that, but I think making the change I described above to _chunk_read() would be better as it would allow you to keep a low timeout (which is desirable) and also not return a short block if you happen to hit a stall.
Something like this completely untested change:
diff -r 61ba9c815170 chirp/drivers/yaesu_clone.py --- a/chirp/drivers/yaesu_clone.py Thu Mar 15 23:41:07 2018 +0000 +++ b/chirp/drivers/yaesu_clone.py Wed Mar 21 16:05:01 2018 -0700 @@ -43,14 +43,24 @@
def _chunk_read(pipe, count, status_fn): + timer = time.time() block = 32 data = "" - for _i in range(0, count, block): - data += pipe.read(block) - if data: + while len(data) < count: + # Don't read past the end of our block if we're not on a 32-byte + # boundary + chunk_size = min(block, count - len(data)) + chunk = pipe.read(chunk_size) + if chunk: + timer = time.time() + data += chunk if data[0] == chr(CMD_ACK): data = data[1:] # Chew an echo'd ack if using a 2-pin cable # LOG.debug("Chewed an ack") + if time.time() - timer > 2: + # It's been two seconds since we last saw data from the radio, + # so it's time to give up. + raise errors.RadioError("Timed out reading from radio") status = chirp_common.Status() status.msg = "Cloning from radio" status.max = count
--Dan
On Wed, Mar 21, 2018 at 4:06 PM, Dan Smith via chirp_devel chirp_devel@intrepid.danplanet.com wrote:
The FT70 has a built in USB interface The Chirp driver is only intended to be used with this, no baud rate is set on it. Using a higher baud rate made a slight difference in speed, But initially thought this might now be causing other issues, so changed back to 38400.
That is very surprising to me. Every other USB serial device I know of honors the baud rate as expected. Unless the radio is really mirroring whatever it can tell the USB serial bridge is set to, but that would be the first such device I've ever encountered.
A quick inspection of the FT70 firmware package from Yaesu shows this is a Renesas microcontroller with a native USB port. In this case it emulates a serial port directly in firmware, rather than a separate USB-serial bridge chip. That's probably a first for an HT, but it's pretty common in the wider microcontroller world.
On devices like that, they just absorb the baud rate changes and pretend to be running at that rate, but it's a no-op inside the microcontroller. Because of this, the actual data rate over the wire can be much faster, since it's only limited by USB data rate and microcontroller speed.
Nicolas, I wouldn't worry about the baud rate changes; just leave it how it was and it should work fine and operate at max throughput regardless. However, changing it also shouldn't cause issues.
-- Brian
A quick inspection of the FT70 firmware package from Yaesu shows this is a Renesas microcontroller with a native USB port. In this case it emulates a serial port directly in firmware, rather than a separate USB-serial bridge chip. That's probably a first for an HT, but it's pretty common in the wider microcontroller world.
Yeah, I meant it'd be the first time I saw an HT that didn't just have a USB-serial bridge chip in it, despite appearing to have "native USB".
On devices like that, they just absorb the baud rate changes and pretend to be running at that rate, but it's a no-op inside the microcontroller. Because of this, the actual data rate over the wire can be much faster, since it's only limited by USB data rate and microcontroller speed.
Nicolas, I wouldn't worry about the baud rate changes; just leave it how it was and it should work fine and operate at max throughput regardless. However, changing it also shouldn't cause issues.
Agreed, especially if this is a reading problem and not a writing one, I wouldn't expect it to matter for stability.
--Dan
**** Thanks for the feedback!
On 21 March 2018 at 23:06, Dan Smith via chirp_devel < chirp_devel@intrepid.danplanet.com> wrote:
The FT70 has a built in USB interface The Chirp driver is only intended
to be used with this, no baud rate is set on it. Using a higher baud rate made a slight difference in speed, But initially thought this might now be causing other issues, so changed back to 38400.
That is very surprising to me. Every other USB serial device I know of honors the baud rate as expected. Unless the radio is really mirroring whatever it can tell the USB serial bridge is set to, but that would be the first such device I've ever encountered.
**** The USB interface is built into the radio, I would need to research further but the baud rate does not effect the radio side of the connection. It enumerates as /dev/cu.usbmodem1D141 basically a USB interface provided by the radio microprocessor. I don't know if any/many other Chirp supported radios have a built in USB interface.
**** Thanks for the feedback! Agreed we need to look at revising the read in yaesu_clone I think this issue effects the FT1D as well
At 38400 or 15200 baud blocks were being returned with no data in them.
So the problem is with _reading_ from the radio, not writing to it? In almost all cases of flaky "it works on my system but not his" cases, it was writing to the radio that was the problem. This is usually because the radio has subtle timing requirements (or a small buffer with no flow control). So, odd thing #2.
**** It seems to write fine, although one to triple check.
Investigated further with Fred and Mark We are losing blocks at the beginning in the Chirp yaesu_clone method.
[2018-03-19 21:35:41,483] chirp.drivers.yaesu_clone - DEBUG: Read 0/65920 [2018-03-19 21:35:41,733] chirp.drivers.yaesu_clone - DEBUG: Read 0/65920 [2018-03-19 21:35:41,986] chirp.drivers.yaesu_clone - DEBUG: Read 0/65920 [2018-03-19 21:35:42,236] chirp.drivers.yaesu_clone - DEBUG: Read 0/65920
I thought the rest of the download was working, hence the puzzle of how
are we short at the end. Which did not make sense if we were only losing blocks at the beginning...
These aren't blocks, they're chunks, and the yaesu_clone code assumes that each chunk of the whole block can and will be read. Sounds like that code should at _least_ assert that it read the length expected, because if it doesn't, it will silently return less data than the block requested. In reality, it should probably be smarter and plan to read the total number of bytes regardless of how many trips it takes, and have a overall timeout to catch the case where we don't hear anything from the radio.
After increasing the size last night it worked and then we realised that
blocks must be being returned with no data during the rest of the download. Slight confusion of the debug info as that shows the running total - not the bytes read for each block - errors will show as an unchanged value not as zero... Loaded it into a spreadsheet this morning and behold we are losing a dozen or so blocks...
Two examples
[2018-03-19 21:18:16,013] chirp.drivers.yaesu_clone - DEBUG: Read
49152/65920
[2018-03-19 21:18:16,263] chirp.drivers.yaesu_clone - DEBUG: Read
49152/65920
[2018-03-19 21:18:16,489] chirp.drivers.yaesu_clone - DEBUG: Read
53248/65920
[2018-03-19 21:18:16,739] chirp.drivers.yaesu_clone - DEBUG: Read
53248/65920
etc. Note the read figure has not changed, so as you said that block
read did not return any data.
Yeah, so what I'm saying about changing the behavior of _chunk_read() above would apply I think.
I submitted a patch to increase the size - for the time being - but it
seems that yaesu_clone needs looking at.
Okay, so I had this backwards because I thought you were talking about uploading not downloading. So I understand why that being larger makes a difference, but it'd be wrong for anyone that doesn't stall at the beginning. I guess the same stupidness in _chunk_read() makes that not completely hang for those people, but...not the right solution :)
chirp sets the time out to .25 secs I changed it to 2 secs and seems to
work.
Yes, and some drivers have to change this because of their radios (although it's always on write AFAIK). You could do that, but I think making the change I described above to _chunk_read() would be better as it would allow you to keep a low timeout (which is desirable) and also not return a short block if you happen to hit a stall.
Something like this completely untested change:
diff -r 61ba9c815170 chirp/drivers/yaesu_clone.py --- a/chirp/drivers/yaesu_clone.py Thu Mar 15 23:41:07 2018 +0000 +++ b/chirp/drivers/yaesu_clone.py Wed Mar 21 16:05:01 2018 -0700 @@ -43,14 +43,24 @@
def _chunk_read(pipe, count, status_fn):
- timer = time.time() block = 32 data = ""
- for _i in range(0, count, block):
data += pipe.read(block)if data:
- while len(data) < count:
# Don't read past the end of our block if we're not on a 32-byte# boundarychunk_size = min(block, count - len(data))chunk = pipe.read(chunk_size)if chunk:timer = time.time()data += chunk if data[0] == chr(CMD_ACK): data = data[1:] # Chew an echo'd ack if using a 2-pincable # LOG.debug("Chewed an ack")
if time.time() - timer > 2:# It's been two seconds since we last saw data from the radio,# so it's time to give up.raise errors.RadioError("Timed out reading from radio") status = chirp_common.Status() status.msg = "Cloning from radio" status.max = count--Dan _______________________________________________ chirp_devel mailing list chirp_devel@intrepid.danplanet.com http://intrepid.danplanet.com/mailman/listinfo/chirp_devel Developer docs: http://chirp.danplanet.com/projects/chirp/wiki/Developers
participants (4)
-
Brian Dickman -
Dan Smith -
nicolas jon pike -
Nicolas Pike