Hi there, I’m just getting started with a Moku:Pro and starting to write a DAQ package for taking long datasets overnight and whatnot.
My understanding is that if I’m writing code to save data for later post-processing, I must first save it as a .li file, which I can currently convert to .csv, .mat, or .npy. As HDF5 is a common storage solution for scientific data, would it be possible to include this as another format?
Thanks!
Hi @slw, sorry for the slow reply but your question took me a little by surprise!
I had written an LI to HDF5 converter several years ago and wanted to check the status of that before I got back to you. Turns out that that output format was removed when we upgraded several performance and other issues a few years back, it had seen very little use. I’ve entered your request to reinstate it in to our system, I agree with you, HDF5 is a very nice format for extremely large datasets and I’ll have to look in to what it would take to get it back! In the meantime, the NPY format is Python-specific of course, but has a lot of the same practical advantages as HDF5 like memory mapping and platform invariance.
Ben.
Hi @bnizette,
Thanks! I’ve been playing around with taking continuous data with our Moku:Pro, where the idea is that we would save, e.g., a minute long dataset, and we would then do offline analysis on that dataset.
Right now, the Moku will save the data to the internal SSD as an LI file. I then need to move it over to whatever computer we will be doing offline analysis on, then convert it to either CSV, MAT, or NPY. When comparing these three formats, CSV is quite large, MAT (via scipy.io.loadmat) has an annoying/strange nested structure, and NPY does not have the metadata associated with the data that was taken.
So, for now, since each of these file formats have some negatives, I would like to convert to HDF5 with a format that I like. One way of doing this is starting with LI → MAT (not CSV to avoid making a large file, and not NPY so I can keep the metadata), read and reformat the metadata, read the data, and save to a new, compressed HDF5 file. But, this isn’t very I/O efficient with the various files being opened, created, closed, etc.
Assuming the LI to HDF5 conversion doesn’t return, would it be possible to have a direct LI file reader in Python? As far as I can tell, the current moku Python package doesn’t have such a function, and it would clean up this workflow quite well, where I could just keep all my data as LI files and directly read them.
Ah yes I’d forgotten that the NPY format doesn’t retain metadata, good point!
The direct file loader is also on our to-do list, I suspect it’ll happen sooner than the HDF5 support but I daren’t give an estimated date. I’ll report back here as we make progress!
Okay, good to hear that’s on the to-do list! Would definitely appreciate the heads-up when that feature is added. I’ll go with the LI → MAT → HDF5 workflow for now, thanks for the info.