Status
Requested
Complexity
8/10
Description
Support binary formatting.
Note that it's a misconception that binary serialization is inherently more compact or faster than other methods; it's actually the unstructured nature of BinaryWriters/Readers which makes them more compact (i.e storing data sequentially in an inflexible way rather than using formatting).
Using binary with formatted data is unlikely to provide any significant gains in performance or reduction in file size over JSON, and in almost all cases we recommend using Compression to achieve the same thing instead as binary formatting is difficult to debug, not editable in a text editor and inflexible:
https://docs.moodkie.com/easy-save-3/es ... ompression
Binary Formatting
-
- Posts: 10
- Joined: Mon Mar 18, 2019 3:12 pm
Re: Binary Formatting
Hello,
Are there any plans to implement this feature? I understand it's quite complex and time consuming, but what about some soft version, like replacing most common types with some short codes? Some two digit code instead of "System.Int32", translating it in input and output from dictionary and put this option into ES3Settings. It would reduce size roughly by 20% in most of the cases, which is nice.
Are there any plans to implement this feature? I understand it's quite complex and time consuming, but what about some soft version, like replacing most common types with some short codes? Some two digit code instead of "System.Int32", translating it in input and output from dictionary and put this option into ES3Settings. It would reduce size roughly by 20% in most of the cases, which is nice.
Re: Binary Formatting
Hi there,
We've currently been experimenting with binary formatting but have decided to put it on hold. The reason for this is because we performed some user research to understand why people were asking for binary formatting.
Generally people have been requesting it for three reasons:
Regarding the second reason: this is the purpose of encryption. Binary formatting only gives the false sense of obfuscation. We might consider adding XOR obfuscation if people find that AES isn't performant enough.
Regarding the third reason: JSON save files can be converted to byte arrays quite easily using ES3.LoadRawBytes.
We then looked at the cons of binary formatting, which we're very aware of as Easy Save 2 revolved around binary formatting. The main cons being:
All the best,
Joel
We've currently been experimenting with binary formatting but have decided to put it on hold. The reason for this is because we performed some user research to understand why people were asking for binary formatting.
Generally people have been requesting it for three reasons:
- Smaller file sizes
- To obfuscate save data
- To work with APIs which require a byte array
Regarding the second reason: this is the purpose of encryption. Binary formatting only gives the false sense of obfuscation. We might consider adding XOR obfuscation if people find that AES isn't performant enough.
Regarding the third reason: JSON save files can be converted to byte arrays quite easily using ES3.LoadRawBytes.
We then looked at the cons of binary formatting, which we're very aware of as Easy Save 2 revolved around binary formatting. The main cons being:
- It's extremely difficult to debug compared to text-based formats
- It's very easy to corrupt. For example, anything which tries to apply an encoding to it will corrupt it, including text editors or incorrectly configured cloud services
- Binary formatting is largely proprietary, so it would be impossible to parse the data using other services, making it much less useful
- It increases the amount of testing required exponentially
All the best,
Joel
-
- Posts: 10
- Joined: Mon Mar 18, 2019 3:12 pm
Re: Binary Formatting
Hello Joel,
Thanks for detailed explanation, I understand that.
But maybe you could still consider my suggestion in post above. I'm asking, because save files in my RPG game go easily above 20MB and I'm quite woried that average player can produce 100+ saves (there are also auto saves during game), which is 2GB+ and that's huge.
I don't save object or anything more complex, mostly only integers, strings, floats, bools, lists and dictionaries. Basically I have to save a lot of variables for 1000+ NPCs. So shortening some strings in saved JSON would make significant difference in size.
PS: I know there is a compression available, but don't want to impact performance. Will test it though.
Thanks for detailed explanation, I understand that.
But maybe you could still consider my suggestion in post above. I'm asking, because save files in my RPG game go easily above 20MB and I'm quite woried that average player can produce 100+ saves (there are also auto saves during game), which is 2GB+ and that's huge.
I don't save object or anything more complex, mostly only integers, strings, floats, bools, lists and dictionaries. Basically I have to save a lot of variables for 1000+ NPCs. So shortening some strings in saved JSON would make significant difference in size.
PS: I know there is a compression available, but don't want to impact performance. Will test it though.
Re: Binary Formatting
Hi there,
Give compression a go as Gzip doesn't impact performance considerably, especially when you use caching. And with larger files you often get some performance back from not needing to write as much data to disk.
With regards to creating shortcuts for type strings, this isn't too difficult to implement at our end. If you private message me your invoice number I can send over a package which does this.
All the best,
Joel
Give compression a go as Gzip doesn't impact performance considerably, especially when you use caching. And with larger files you often get some performance back from not needing to write as much data to disk.
With regards to creating shortcuts for type strings, this isn't too difficult to implement at our end. If you private message me your invoice number I can send over a package which does this.
All the best,
Joel
-
- Posts: 10
- Joined: Mon Mar 18, 2019 3:12 pm
Re: Binary Formatting
Thank you Joel, but no need to sending me an update as I am not in hurry (long development ahead). Can wait for some future update.