Performance Issues (Multithreading?)

[Jason]

Hi there,

I've been thoroughly enjoying easy save thus far. It has worked wonders for the last game I made.

Now, though, I'm making a voxel-based game and figured I could use Easy Save to save chunk data to files. I wrote some optimizations so that it only saves modified block data, and not every block that makes up the chunk (16,000+ bytes).

However, of course, there's the situation where players may modify a whole lot of blocks on a particular chunk. I haven't tested on mobile, but I imagine this is going to have a performance hit trying to load so much data for chunks from a file (since I plan on using it for mobile). (My terrain is infinite, so I'm loading it on the fly right now).

As I'm just starting voxel engine building, I'm not incredibly experienced yet and learning as I go. However, I was thinking that multithreading would be a great solution to these issues. So I have multithreading for actually generating mesh data for chunks, but I can't seem to use it with easy save due to "get_isWebPlayer" requiring the main thread.

So I'm guessing multithreading isn't supported? Is there an easy way for me to work around this issue?

[Joel]

Hi Jason,

As the Unity Engine is synchronous, we've chosen to make it synchronous too. If enough people request it however, we may consider adding some asynchronous methods of our own (the main bulk of ES2 has to be synchronous because it can't operate outside of the main thread, but saving/loading data to and from files could be done asynchronously).

The only way around this at the moment would be to write your own methods to read the data from a file as a byte array asynchronously, and then use an ES2MemoryReader (an undocumented class, but works the same as an ES2Reader) to read the data from the byte array. For example:

// Get the bytes from the file async using your own method.
// This might be how it would be called from within the main thread.
AsyncClass asyncClass = new AsyncClass();
yield return asyncClass.GetBytesAsync(Application.persistentDataPath+"/myFile.txt");
byte[] fileBytes = asyncClass.bytes;

Chunk[] chunks;

using(ES2Reader reader = new ES2MemoryReader(fileBytes))
{
    int chunkCount = reader.Read<int>();
    chunks = new Chunk[chunkCount];

    // You could use coroutines to spread your Load calls across multiple frames too.
    // This example loads 1000 chunks per frame.
    for(int i=0; i<chunkCount; i++)
    {
        chunks = new Chunk();
        chunks.position = reader.Read<Vector3>();
        chunks.id = reader.Read<uint>();
        // etc, etc

        if(i % 1000 == 0)
                yield return null;
    }
}

There's also an ES2MemoryWriter which works in the same way as an ES2Writer, but it also has a storedBytes variable which is set after calling save. You could use this as follows:

byte[] fileBytes;

using(ES2Writer writer = new ES2MemoryWriter())
{
    writer.Write(chunks.Length);

    for(int i=0; i<chunks.Length; i++)
    {
        writer.Write(chunks.position);
        writer.Write(chunks.id);

        if(i % 1000 == 0)
                yield return null;
    }

    // Now call save and get the bytes from our ES2MemoryWriter.
    writer.Save();
    fileBytes = writer.storedBytes;
}

// Now call your method to write the bytes to file asynchronously.
yield return WriteBytesAsync( fileBytes, Application.persistentDataPath+"/myFile.txt" );

This should give you the general idea as to how this would work. Also if you're not doing so already, use ES2Reader and ES2Writer sequentially; it's much, much faster than using tags.

I sure hope this helps. Asynchronous methods have been added to the To Do list, so they should make it's way into Easy Save eventually anyway.

Just in case you need them, here are the constructors for ES2MemoryWriter and ES2MemoryReader:

ES2MemoryWriter()
ES2MemoryWriter(ES2Settings settings)
ES2MemoryReader(byte[] bytes)
ES2MemoryReader(byte[] bytes, ES2Settings settings)

Edit: I've just realised that ES2MemoryReader is currently sealed. It'll make it accessible in v2.42, and I'll also change it so that it comes up in code completion.

All the best,
Joel

[heineken1812]

Hello,

As of version 2.48 there is still no access to ES2MemoryWriter/Reader. On Windows Store/Phone you have to be able to deal with File I/O asynchronously, as it's recommended (truly required).

Even saving a simple int with ES2 causes a 200ms spike on every WP I can test with, including my Lumia 1520.

IMO this make ES2 just not ready for Windows Store or Phone until it can properly handle async File I/O.

And without access to the ES2MemoryWriter it's not even possible to integrate our own DLL plugin to fix this.

[Joel]

Hi there,

ES2MemoryWriter/Reader were included in v2.42 to v2.45, but Easy Save was restructured in v2.46 and ES2MemoryWriter/Reader was merged into ES2Writer/Reader.

You can implement your own functionality as follows:

public static void Save<T>(string tag, T data)
{
    byte[] bytes = null;

    using(ES2Writer writer = new ES2Writer(new ES2Settings()))
    {
        // You can have multiple writes here if saving multiple pieces of data.
        writer.Write(data, tag);
        bytes = writer.stream.ReadAllBytes();
    }

    // *** Use your own code here to async store the data ***
}

public static T Load<T>(string tag)
{
    byte[] bytes = null; 
    // *** Use your own code here to async load the data ***

    using(ES2Reader reader = new ES2Reader(bytes, new ES2Settings() ))
    {
        // You can have multiple reads here if necessary.
        return reader.Read<T>(tag);
    }
}