Accessing Dive Info on the UEMIS SDA

[Scubacastor]

btw the sync with the UEMIS portal is back to normal. (more dive and divelog.xml to get !!)

 

[BB1]

Weird - seems to work here. Just in case, I put a new zip file at the same server: http://65.23.157.151/uemis.zip

That zip file has been tested...

Yep that new .zip file works fine!!

Today I did the real test. After a couple of great dives in Hoodsport, WA, I disconnected my laptop from the Internet (to make absolutely sure that nothing in this process needed access to the site), connected the UEMIS, extracted and parsed the dives, imported them into Diveinglog and everything came out perfectly.

Thank you, UEMIS - I don't need your stupid portal with its annoying Flash based dive log software anymore.

Did all information parse OK or are these pieces of information still missing?

diveNNN.xml:
- format of floats is not always consistent
- divesite information is missing (not extracted from SDA)
- gear information is missing (not extracted from SDA)

divelogNNN.xml:
- date_time item isn't parsed
- various pressures are mis-calculated
- mod and tcd in the gas section aren't parsed

 

[Chiung]

Looks like lots of progress - awesome!

soooooo.....

When can mere mortals like me expect a plug and play option? I'm ready to buy if it's ready for prime time. What would really be nice is a new thread with some sort of walk-through with some illustrations and screen captures of the process.

Thanks to all who have contributed

 

[Scubacastor]

Yep some parsing still requires some development/check.

It appeared that the location, gear, contact data are not written in the dive data file other than as indices (divespot number) or name of contact, therefore these types of data have to be extracted seperately in order to complete the information needed in dive.xml files.

Divelog.xml is pretty much self contained in terms of infos but the data coding is sometimes very strange. Parsing correctly the date/time info is for now a mystery for this file. On my script, I read the time in the header section of the files, convert it to the weird format UEMIS use and then in the post-dive info do the same again with the dive time added.

About the gas section, it took me probably 10 hours to understand how to read correctly the hex data for tank size, mod, tcd, reserve, limit and I am glad to say I finally solved the puzzle. I'll provide the info tomorrow to the various people involved in this project so it should be corrected quickly.

Everything except time/data, seems to be parsed correctly in my php script so far for divelog.xml . I am writting the script for dive.xml currently.

 

[dirkhh]

About the gas section, it took me probably 10 hours to understand how to read correctly the hex data for tank size, mod, tcd, reserve, limit and I am glad to say I finally solved the puzzle. I'll provide the info tomorrow to the various people involved in this project so it should be corrected quickly.

Awesome - I look forward to seeing that and will add it to my parser.

I am writting the script for dive.xml currently.

I believe that my tool already parses most of what's in there (with the exception of the indices into other databases...). I'm curious to see if you found additional detail that I didn't notice.

 

[dirkhh]

Did all information parse OK or are these pieces of information still missing?

Still a few things missing - but enough is parsed to be able to import your dives into divinglog - which to me was the major goal. To prove that we can import dives (and keep records) without needing the UEMIS web server.

As you appear to have gotten the tool to work, can you provide SDA and matching xml files for those of us continuing to work on the tools? I had more than 30 people download the tools, but no one so far has provided sample data - which is what we need to improve our parsing.

Ideally we'd want the matching diveNNN.SDA, divelog.SDA, diveNNN.xml, divelogNNN.xml from a number of dives and the matching xml files from the UEMIS server. I call those uemis.xml in my Linux/Mac scripts - you can also manually download them from the website if you are on Windows. The corresponding XXXX number for the web site can be found in the diveNNN.SDA file:

The NNN numbering my tool uses is the dive number as seen by the SDA. Once you upload a dive to the UEMIS website it uses a unique ID for each dive that is uploaded. after you upload the dive information that corresponding number is stored in the diveNNN.SDA file. It looks like this:

<val key="remote_object_id">
<int>0</int>

Where '0' is replaced by the ID used on their web site - so if you find 654321 there then the files on the webserver are

http://www.uemis.com/en/sda_logbook/data_src/654321/dive
http://www.uemis.com/en/sda_logbook/data_src/654321/divelog

If some of you could package these six files per dive together in a zip file and post here (they compress really well) or send them to me in email that would really help us to improve the tool.

 

[Chiung]

Still a few things missing - but enough is parsed to be able to import your dives into divinglog - which to me was the major goal. To prove that we can import dives (and keep records) without needing the UEMIS web server.

As you appear to have gotten the tool to work, can you provide SDA and matching xml files for those of us continuing to work on the tools? I had more than 30 people download the tools, but no one so far has provided sample data - which is what we need to improve our parsing.

Ideally we'd want the matching diveNNN.SDA, divelog.SDA, diveNNN.xml, divelogNNN.xml from a number of dives and the matching xml files from the UEMIS server. I call those uemis.xml in my Linux/Mac scripts - you can also manually download them from the website if you are on Windows. The corresponding XXXX number for the web site can be found in the diveNNN.SDA file:

The NNN numbering my tool uses is the dive number as seen by the SDA. Once you upload a dive to the UEMIS website it uses a unique ID for each dive that is uploaded. after you upload the dive information that corresponding number is stored in the diveNNN.SDA file. It looks like this:

<val key="remote_object_id">
<int>0</int>

Where '0' is replaced by the ID used on their web site - so if you find 654321 there then the files on the webserver are

http://www.uemis.com/en/sda_logbook/data_src/654321/dive
- uemis - THE UNDERWATER REVOLUTION

If some of you could package these six files per dive together in a zip file and post here (they compress really well) or send them to me in email that would really help us to improve the tool.

I've got about 80 dives in myuemis. How do I get them from there to here?

 

[dirkhh]

I've got about 80 dives in myuemis. How do I get them from there to here?

Depends on the OS you use. If you are on Linux or a Mac then the script that I posted can automate the process. If you are on Windows, it's all manual.
We don't need all 80 dives. Pick a few that are special. A gas change? An odd tank (HP190? LP65?)? Doubles? Deco? Or just a couple that you like.
Extract the data from the SDA (using myuemis.jar - the command line is earlier in this thread - use diveNNN.SDA and divelogNNN.SDA as output file names).
Run the parser against this (calling it with diveNNN.SDA as argument). This creates diveNNN.xml and divelogNNN.xml.
Finally, look at the diveNNN.SDA file, find the "remote_object_id" and download the corresponding xml data from the uemis website.
Now post all six files here.

Thank you.

 

[Koni]

Hi all,
I was 2 weeks on diving vacation and as I come back all we waited so long for from UEMIS seems to be solved now, great job!!!

In advance I spent some time to reverse engineer the binary data out of the UEMIS sync log files. Now I'm able to interprete >90% of the pre and post header data as well as the profil data including the warning bits. Further I did some test dives using the multigas setups and also generated an number of warnings and alarms.
I think I could bring in some more info on how to interpret the binary data. Is there somewhere a common place where all this stuff is collected and stored?

Yesterday I upgraded my Diving Log software and downloaded and imported all my dives from the UEMIS server. All worked great without any problems.
The only bug I found are the depth values in the profil as well as in the header data. The UEMIS SDA stores the water pressure and not the depth values. I think to be fully correct the values have to be multiplied with a factor of ~1.015 for fresh water, resp. ~0.98 for salt water.

Thanks,
Koni

 

[dirkhh]

Hi all,
In advance I spent some time to reverse engineer the binary data out of the UEMIS sync log files. Now I'm able to interprete >90% of the pre and post header data as well as the profil data including the warning bits. Further I did some test dives using the multigas setups and also generated an number of warnings and alarms.
I think I could bring in some more info on how to interpret the binary data. Is there somewhere a common place where all this stuff is collected and stored?

I would love to see what you've learned in addition to what I already have. Especially the warnings are completely unknown to me at this point.
In typical open source fashion the code is the documentation of what we know. I have posted my code here a couple of days ago. Since then Scubacastor has figured out the encoding of the gas data and sent that to me last night, so I hope to add this to my parser later today - I'll post an update here when I have that.

Maybe we can start a wiki page somewhere to help collect the missing pieces?

Yesterday I upgraded my Diving Log software and downloaded and imported all my dives from the UEMIS server. All worked great without any problems.
The only bug I found are the depth values in the profil as well as in the header data. The UEMIS SDA stores the water pressure and not the depth values. I think to be fully correct the values have to be multiplied with a factor of ~1.015 for fresh water, resp. ~0.98 for salt water.

I guess this is something that should be addressed in the divinglog importer, not in my parser, as I want the parser to create fundamentally the same output as the XML files from the UEMIS website. divinglog, is this something you could look into?