Un-bricking a broken API

Moku model: Moku:Pro (HW 3.0, MokuOS 4.1.2, firmware v. 635.0, API server v. 4.1.2.1)
Operating system: Debian Linux
Software version: mokucli v. 4.1.1.1, downloaded instrument bitstreams v. 4.1.2, Python package ‘moku’ from PyPi v. 4.1.2.1.

Bug or support request description:

We control the Moku via the API in various configurations in a remote lab. Most frequently, it is configured in the MIM regime with one of the slots being a LaserLockBox, with which our scripts communicate through the API to control cavity auto-relocking etc. Connections are usually performed with persist_state=True parameter.

It works most of the time, but occasionally (≈once per day) the API starts returning an error ‘Error fetching properties. Please try again.’ for any connection attempt. I am not sure under what particular situation it appears, but usually it seems to have something to do with loading/saving .mokuconf snapshots via the API (sometimes, but not always, it is preceded by another error – '“[oatpp::data::stream::FileOutputStream::FileOutputStream(filename, mode)]: Error. Can’t open file.”). Once the error appears, it is consistent when trying to connect to the Moku from different machines, either from the Python API, from the web API, or with the mokucli. However, the GUIs for Windows/Mac work, and the web interface is available (but the API returns the error I mentioned above to any request, including a request to reboot). The API function can be restored to normal operation by physically rebooting the Moku.

We were unable to find a way to reboot a Moku from a GUI or from the web interface. This is quite annoying, because the device is at a remote site: once the API breaks, there seems to be no way of bringing it back to work other than have someone go and physically interact with the power button on the device.

I guess our problem can be reduced to three questions:

  1. What actions could lead to this error?
  2. Is there a way to fix it without rebooting the device?
  3. Is there a way to reboot the device with a broken API (i.e. without access to the .reboot() method) by other means, e.g. through the GUI or the web interface? We can also power cycle the mains remotely, but it leaves the Moku switched off until the power button is pressed – perhaps it could be configured to restore the last power state?

Typical error stack:

$ mokucli command Moku reboot --ip=<our_moku_ip> --force-connect -v
✗ Error executing command: An unexpected error occurred during initialization: ['Error fetching properties. Please try 
again.']

Full traceback:
Traceback (most recent call last):
  File "moku/__init__.py", line 138, in __init__
  File "moku/__init__.py", line 272, in claim_ownership
  File "moku/session.py", line 23, in func_wrapper
  File "moku/session.py", line 204, in resolve
  File "moku/session.py", line 166, in _handle_error
moku.exceptions.MokuException: ['Error fetching properties. Please try again.']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "mokucli/api_command.py", line 172, in execute_command
  File "moku/__init__.py", line 175, in __init__
moku.exceptions.MokuException: An unexpected error occurred during initialization: ['Error fetching properties. Please try again.']

Hello @admitrieff ,

Thank you for reaching out to Liquid Instruments. Please reach out to support@liquidinstruments.com for more comprehensive support.

Hi, why was this discussion moved to a private channel?

My lab encounters a similar issue (I forget if the error message is the same) and we reboot the moku each time. It could be useful to read about this issue.

Hi @jephwack ,

For more complex issues with extended correspondence we typically like to handle those cases in email to protect customer IP and discuss meeting details. Feel free to post your issue to the forum or reach out to support@liquidinstruments.com for help with your issue. Thank you!

-Dylan

This issue was related to load_configuration/settings(); the issue was fixed in MokuOS 4.2.