Feature #473
closedNetwork protocol: 8bit and 16bit delta array lengths
0%
Description
Recently delta protocol array indices were bumped from 8bit to 16bit, to support arrays with more elements. However, this means that transferring any array headers now take twice as much space.
Maybe we should have two array types? One with 8bit indices, and one with 16bit indices. E.g., "uint short_array[less_than_254_elems]8;" & "uint long_array[more_than_253_elems]16;" Or maybe have the 8bit array the default, and make an array 16bit one only by specific marking?
Files
Updated by Alina Lenk 7 months ago
- Category set to Bootstrap
Has this had an actual noticeable impact, or are we prematurely optimizing? As much as it pains me to admit it, there does come a point where performance optimization might not be worth making the code more complex. Most of our network traffic is compressed, so I'd hope if there's enough extra null bytes to be a significant concern, compression would mitigate that again.
If we're gonna do this, I'd prefer not to complicate the packets.def format; and more importantly, not to specify the index size limit in two places, so whenever we change a constant used as an array's declared/maximum size, we also have to check whether we now have to change a bunch of index sizes.
We could determine the required index type statically by comparing the declared max size to MAX_UINT8; or we could even do it dynamically (if there were any diff arrays with non-constant size), since at the time we send or receive a deep-diff array, we already know the length of that array, and thus the max index. Though doing it dynamically would make the generated code a lot more complex.
(If and when we're gonna be deep-diffing vectors, we'll only have the dynamic option, naturally.)
Updated by Alina Lenk 7 months ago
Alina Lenk wrote in #note-1:
Though doing it dynamically would make the generated code a lot more complex.
Turns out after some testing, the simpler way I thought would work in the static case doesn't actually work without extra macro ugliness. So if we do this it's gonna result in a bunch of if-elses (rather than just a single one at the start) either way; just in the static case, they're preprocessor if-elses rather than runtime ones. I've also realized we can simplify some of the logic by using the actual array size as a sentinel value, rather than MAX_UINT8/MAX_UINT16.
Updated by Alina Lenk 7 months ago
- File 0002-generate_packets.py-use-8bit-array-diff-indices-when.patch 0002-generate_packets.py-use-8bit-array-diff-indices-when.patch added
- Status changed from New to In Review
- Target version set to 3.3.0
Voila. I did reformat and move around some parts of it to make that part of the code more readable, I think it's okay now.
Updated by Marko Lindqvist 7 months ago
Should we have this also in S3_2, as the original change to 16bit array lengths was applied there?
Updated by Alina Lenk 7 months ago
- File 0001-generate_packets.py-use-8bit-array-diff-indices-when.patch 0001-generate_packets.py-use-8bit-array-diff-indices-when.patch added
- Target version changed from 3.3.0 to 3.2.0-npf
Marko Lindqvist wrote in #note-4:
Should we have this also in S3_2, as the original change to 16bit array lengths was applied there?
Probably, yeah.
Updated by Alina Lenk 7 months ago
- File main-generate_packets.py-use-8bit-array-diff-indices-when.patch main-generate_packets.py-use-8bit-array-diff-indices-when.patch added
- File S3_2-generate_packets.py-use-8bit-array-diff-indices-when.patch S3_2-generate_packets.py-use-8bit-array-diff-indices-when.patch added
Fixed a typo (MAX_UNIT8 instead of MAX_UINT8).
Updated by Alina Lenk 7 months ago
- File main-generate_packets.py-use-8bit-array-diff-indices-when.patch main-generate_packets.py-use-8bit-array-diff-indices-when.patch added
- File S3_2-generate_packets.py-use-8bit-array-diff-indices-when.patch S3_2-generate_packets.py-use-8bit-array-diff-indices-when.patch added
Fixed an off-by-one error in JSON mode (receiving side was always ignoring the first diff element).