UI Simple

Revision as of 02:43, 23 August 2010

MIOS is a lightweight home automation system. The 'brain' of the MIOS software is the back-end, the engine, which runs stand-alone on a variety of internet-connected devices, such as PC's, Mac's, Wi-Fi access points, and dedicated home automation gateways. MIOS also includes a portal at mios.com, which acts as secure relay to MIOS systems that may be behind firewalls. Users can register for an account at mios.com, and that account can be linked to one or more MIOS systems to provide the user remote access to his MIOS system from anywhere. It is easy to control a MIOS system with simple http get's (normal internet requests). The URL you will open is generally data_request?id=xxx, where xxx is some sort of request or control command.

This document describes how to create a simple user interface to control a MIOS system using the simplified lu_sdata (LuaUPnP Simple Data) request. This document describes a simple control-only user interface, meaning it let's the user run scenes and control devices, but does not provide any means to change configuration or do advanced tasks. Whatever device is running the user interface (cell phone, web page, television, etc.) will be referred to as the "controller", and the MIOS engine, or system, that is being controlled by the controller is the "engine".

You can test out all the commands a normal web browser. For example, if your engine is on the same local network as your web browser, and your engine has the IP address: 192.168.2.150, you can view the status of all scenes and devices by opening this link in your browser: http://192.168.2.150:3480/data_request?id=lu_sdata and if you want to turn on device #5 you open this link in your browser: http://192.168.2.150:3480/data_request?id=lu_action&DeviceNum=5&serviceId=urn:upnp-org:serviceId:SwitchPower1&action=SetTarget&newTargetValue=1

The user interface has only two screens, or modes: 1) Basic setup which only consists of identifying the engine the user will control, and 2) Normal usage which consists of running scenes (pre-defined groups of commands), and controlling devices.

Mode 1: Basic setup and locating the engine

The first thing the user interface should do is find the engine on the internet. There are 2 different ways to control an engine:

1) Directly, using an IP address, where the engine is either on the same local area network as the controller or has a static IP or port forward that's publicly accessible from the internet, such as in the example links above.

2) Through one of the MIOS secure forward servers, which acts as a relay to an engine that may be behind a firewall. The URL is identical except that instead of using the IP you use a mios forward server followed by the /username/password/serial_number, where the username/password are from the user's mios.com account that he linked to his engine, and serial_number is the unique id of the engine. So, assuming you want to turn on device #5 on engine #10266 and the user linked it his mios account with the username "john" and password "tokyo", you would open this URL (note it's identical to the URL above): https://fwd2.mios.com/john/tokyo/10266/data_request?id=lu_action&DeviceNum=5&serviceId=urn:upnp-org:serviceId:SwitchPower1&action=SetTarget&newTargetValue=1

NOTE: Because method #2 is much slower than method #1 when both the engine and the controller are on the same network, it is generally preferred to use method #1 when the controller is in the home, and method #2 when the controller is away from the home and needs to talk to the engine through the homeowner's firewall. If you are putting your UI in something that is only on a local network, like a TV, and you do not want to give the user the ability to control an engine outside the home, you may only implement method #1. If the UI is running on something that a mobile phone that has NO wi-fi and can only connect through the mobile network, then you may want to implement method #2 only. Normally, though, a user interface should be able to use both method #1 and method #2, and this is how we spec it for our UI's.

When the controller first starts, display this:

Enter your mios.com username: [__input_box__] [go]
__I don't have a mios.com account or I want to specify the IP address.__

Where [go] is a button, [__input_box__] is for the user to type in the username, and the __I don't have... is a link, or button, or similar. If the user clicks the 'I don't have' link, display this:

Enter the IP: [__input_box__]  [go]

When the user clicks 'go' attempt to the open this url: http://ip:3480/data_request?id=lu_alive (substitute the actual IP), and if you get back an "OK" in the response, store the IP address in your controller locally (ie a conf file, registry, etc) and continue to the next step. If you don't get an OK, display an error and go back.

If the user supplied a mios.com username, open this URL: http://sta1.mios.com/locator_json.php?username=user (substitute 'user' for the actual username). This will return a list of all the engines you can control, both with method #1 and with method #2, meaning you will see both engines on the local network which may or may not be tied to the user's mios.com account, and you will see engines tied to the mios.com account which may or may not be on the local network. As with most of the requests, the returned data is in JSON format. The data you get back is not formatted with new lines and spaces. If you want to be more human readable for debugging, copy/paste the results into http://jsonlint.com and it will format it nicely. The controller needs to have a json library so it can parse the json responses. The data you get back will be like this:

   "units": [
       {
           "serialNumber": "10266",
           "FirmwareVersion": "1.1.1052",
           "ipAddress": "192.168.2.117",
           "name": "skyvera",
           "users": [
               "skyvera",
               "aaronb"
           ],
           "active_server": "fwd2.mios.com",
           "forwardServers": [
               {
                   "hostName": "fwd2.mios.com",
                   "primary": true
               },
               {
                   "hostName": "fwd1.mios.com",
                   "primary": false
               }
           ]
       },
       {
           "serialNumber": "8035",
           "FirmwareVersion": "1.1.1047",
           "ipAddress": "192.168.2.116",
           "users": [
               "ovidiu",
               "alfonsomios",
               "aaronb",
               "mrhtn"
           ],
           "active_server": "fwd1.mios.com",
           "forwardServers": [
               {
                   "hostName": "fwd1.mios.com",
                   "primary": true
               },
               {
                   "hostName": "fwd2.mios.com",
                   "primary": false
               }
           ]
       },
       {
           "serialNumber": "10516",
           "FirmwareVersion": "1.1.1047",
           "ipAddress": "192.168.2.23",
           "users": [
               
           ],
           "active_server": "fwd2.mios.com",
           "forwardServers": [
               {
                   "hostName": "fwd2.mios.com",
                   "primary": true
               },
               {
                   "hostName": "fwd1.mios.com",
                   "primary": false
               }
           ]
       }
   ]

units is an array of JSON objects, one representing each engine. If the tag ipAddress exists for an engine, that means it's available on the local network and can be controlled locally with method #1, otherwise the ipAddress tag will not exist. The users array is a list of all the mios.com usernames which have access to this engine. So, if the username you passed on the locator_json.php URL is also in the users array, then the engine can be controlled remotely with the username, using whatever server is listed in the "active_server" tag. The forwardServers tag lists the primary server for remote access and one or more backups. If the tag "name" exists, that is a name which the user assigned to his engine.

The next screen in the UI should say "What MiOS engine do you want to control?" and then display a list of all the engines with both the serialNumber and name if it exists. You should have 2 icons next to each for 'remote' and 'local' access. In the above example, assuming the username I passed in is "skyvera" 10516 will have the 'local' icon only, since there are no users, and for 8035 it will also be 'local' only because skyvera is not one of the allowed users. For 10266 display both the 'remote' and 'local' icon. At the bottom of the page you can have a legend:

[R icon] - You can access this MiOS engine from anywhere in the world over the internet using your mios.com account

[L icon] - You can access this MiOS engine locally on your home network without going through your mios.com account

And also a multiple choice asking the user if he wants: Automatic connection, Local connection only, or Remote connection only.

Let the user pick the engine he wants to control and store in the controller's local storage the username supplied by the user, and from the JSON file for whatever engine the user picked store the contents of serialNumber, ipAddress, active_server, and store the list of servers in forwardServers.

At this point display:

 mios.com password: [__input_box__]
 [ ] Store my password so I don't have to enter it each time
 [go]

If the user checks the box, store the password. When the user clicks go, if you stored an ipaddress, test it with:

http://ip:3480/data_request?id=lu_alive

and confirm you get an OK. Then test the password with:

https://xxx.mios.com/username/password/serial/data_request?id=lu_alive and substitute xxx.mios.com for the active_server, and substitute the actual username/password/serial. Again, confirm you get back an OK.

If you do not get an OK for both (or the 2nd one if there was no local ip address), display an error and go back:

Unable to connect.  Please check your password.  [ok]

Mode 2: Normal operation

Now that you have stored the connection information (ip address, etc.), you can start the main application. From now on, whenever the user starts the UI, it should go straight into the main application and not ask him for the connection information again, although within the normal operation the user has the option of returning to setup.

The use can choose to control devices and scenes. A device is a light switch, a thermostat, a door lock, etc. Some devices may not have any control options, meaning the user cannot do anything with them, and they are only there to show status, such as a motion sensor or a temperature sensor. Scenes simply groups of actions the user has pre-defined, such as "Go to bed", which may be a scene that turns off all the lights.

The user is able to create rooms and sections to organize the devices and scenes. Devices and scenes can be assigned to a room, and each room can be assigned to a section. Only very large home with many rooms divide the rooms into multiple sections, such as "West Wing", "3rd floor", etc. By default there is only 1 section and all the rooms belong to it.

Devices and rooms may or may not be assigned to a room. Small installations with only a handful of devices typically do not create rooms, and all the devices and sections are left unassigned. When the user does create rooms, he does not need to assign all devices and sections to a room, so some may remain as unassigned.

Each device has a category, such as 'dimmable light', 'thermostat', 'door lock', and so on.

So, a typical top-level menu may have the following options:

Setup to return to Mode 1 and choose a new engine to connect to. If the user interface has a menu or option button the Setup can be there so it does not occupy space on the main menu.

Scenes This lists all the scenes. First list the scenes not assigned to a room, then list the scenes grouped by the room they are assigned to.

Lights, Thermostats, etc. Next the controller lists the categories so the user can just straight to 'thermostats' and see what thermostats, lights, etc. are in the house. The lu_sdata request will provide you with a list of the categories that are actually used in the system.

Devices This lists all the devices that are not assigned to a room

Living Room, Bedroom, etc. Next is a list of all the rooms, broken down by section. Choosing a room lists the scenes in that room, followed by the devices.

Therefore if a device is assigned to a room, it will appear in 2 places, in both the room and in the device's category. The same is true for scenes. Here is a typical top-level menu:

If the user choose a room that has 2 lights and a thermostat in it, following is what the sub-menu looks like:

The name of the device or scene is the primary identifier along with an icon that depends on the device category. Each device has control options that are specific to the device's category, like 'on' and 'off' for binary light switches, and 'on', 'off' and a slider bar for dimmable light switches. The current status of the device should be indicated both by the status of the icon (light bulb yellow if it's on, gray if it's not), as well as by highlighting the control button corresponding to the status, for example, the 'on' button should be highlighted if the device is on.

Additionally, there should be a space in the device's control or scene's control that is reserved to indicate the 'state' of the device or scene. All devices and scenes can have 1 of the following 4 states: none (nothing is going on with the device or scene), pending (it's actively being controlled), success (the last action performed was successful), error (something is wrong with the device). And, for both scenes and devices there needs to be a way to display a short comment. For large controllers, like a TV, this may mean displaying the comments on the UI whenever they exist. For small displays, like mobile phones, it's possible to have the user touch the device to see more details, such as the comments.

Software architecture: Two threads with a polling loop

Typically the software for the controller is written with 2 separate threads: 1) the back end: a background poll loop that continuously polls the engine for changes and which updates an object-oriented class structure containing the list of sections, rooms, scenes and devices, as well as the state of the engine (ip address, running state, etc.), and 2) the front end: the user interface which displays the data retrieved by the poll loop.

These should be separate as much as possible with separate classes. That way it's possible to reskin the user interface at some point without having to rewrite the poll loop. The poll loop has no user interface and simply runs as a separate thread and just continuously polls the engine. The user interface should be separate from the poll loop so it can be replaced easily.

Both the back end and front end need to access the same object oriented class structure and need to be able to operate completely independently, so separate threads and mutex's are required.

Just like devices and scenes can have 4 different "states", the engine itself also has the same 4 states: none, pending, success, error.

As mentioned, the back end (polling loop) creates a class structure which the front end renders, and one of the things in the class structure is the state of the engine. This should be stored as a signed integer, and the initial value will be -2, which always means 'not connected'. When the front end (the user interface) starts up and sees the engine state is -2 (not connected) it should display a 'Please wait. Connecting...' message.

lu_sdata: The polling loop

The back end is simply a polling loop that continuously requests the "lu_sdata" data_request from the engine. This is done by fetching the following URL: http://ip/data_request?id=lu_sdata

As mentioned in the beginning, you can use the IP address of the engine directly and always use port 3480, like this:

http://192.168.2.117:3480/data_request?id=lu_sdata

or you can connect remotely through the forward servers, like this:

https://fwd2.mios.com/john/tokyo/10266/data_request?id=lu_sdata

You should always connect directly whenever possible. In addition to storing the state of the engine in a shared variable, the back end should also store in a variable if it is connected directly or remotely through the forward server, and the user interface will display an icon indicating if it is connected directly or remotely. If the direct connection fails, you should automatically switch over to the remote connection. Once the network settings have changed, for example, a mobile phone has switched from 3G to Wi-Fi, try again the direct connection.

To make development easier, if you are using Firefox, which has a built-in xml parser, you can add an "output_format=xml" to the URL and the engine will convert the json to xml, which Firefox renders nicely. This is only for testing. Your actual controller must always retrieve everything in the native json format.

For example, MIOS has an engine available to outside developers on a static IP (76.168.224.30). The following URL will show the lu_sdata reformatted as xml in Firefox:

http://76.168.224.30:3480/data_request?id=lu_sdata&output_format=xml

You can use this for developing your user interface too, just drop the &output_format=xml and if you want to format the JSON nicely, visit jsonlint.com.

Note the top-level object contains these tags:

full="1" loadtime="1282441735" dataversion="441736333" state="-1" comment=""

full is either 0 or 1, and 1 means lu_sdata has returned the complete list of sections, rooms, devices and scenes. If fill is 0, the lu_sdata is only returning the devices and scenes which have changed since the last request. On subsequent calls to lu_sdata you will the loadtime and dataversion back on the URL, like this:

http://76.168.224.30:3480/data_request?id=lu_sdata&loadtime=1282441735&dataversion=441736333

The state is the state of the engine. It is the same as the state of the devices and scenes. See "List of states" below to see what the numbers mean. If the state is not 'none', the main menu should display at the top a red, green or blue icon (for error, success or pending states) and show the user what is in the comment tag.

Next are all the device categories listed in an array. Each category has a name and an id tag, like this: name="Thermostat" id="5". You will only see the categories for devices that exist in the MIOS installation. Always use the id for internal reference; the name changes based on the user's language.

Next are all the sections listed in an array. Each section has a name and an id tag, like this: name="My Home" id="1" Again, in most homes, there is only 1 section, and, if there is only 1 section, you do not need to show this to the user.

Next are the rooms. Again, they have a name and an id, and also a section which the room is in.

Next are the scenes. Scenes also have a name and an id. They also have an 'active' of either 0 or 1. If the active is '1', that means the scene is currently active, meaning all the devices are presently in the state the scene will set. So if the scene is 'go to bed', and it turns off all the lights, and if all the lights are currently off, 'active' will be 1 for the scene whether or not the user recently ran the scene. Scenes also have a state and comment (see state list below). The back end (polling loop) actually doesn't do anything with the active, state or comment other than store then in shared objects or variables that the front end (the user interface) uses to render the controls for the user.

Next are the devices. They also have the name, id, state and comment, plus they have a category. Additionally, they have extra tags which depend on the category of device. Light switches, for example, will have a 'status' tag and a value of 0 or 1 indicating if the light is off or on. Thermostats also have a 'temperature' tag. See the 'categories' section below.

The polling loop continues to repeat indefinitely as long as the application is running. Each time it passes the loadtime and dataversion of the prior request. Additionally, you should add a timeout argument to the url. This is the number of seconds that the URL will block waiting for a change. For example, this URL:

http://76.168.224.30:3480/data_request?id=lu_sdata&loadtime=1282441735&dataversion=441736333&timeout=60

Will block for a maximum of 60 seconds if no devices or scenes have changed since the loadtime=1282441735&dataversion=441736333. The timeout should be as large as is reasonable to minimize the amount of traffic between the controller and the engine, but small enough to ensure the socket libraries in the controller do not time out and exit with an error before the 'timeout' number of seconds has passed.

Additionally you should add a minimumdelay to the argument which is the number of millseconds that the engine will block the request even if there are changes. For example:

http://76.168.224.30:3480/data_request?id=lu_sdata&loadtime=1282441735&dataversion=441736333&timeout=60&minimumdelay=2000

means that the request will block for up to 60 seconds and if a change occurs after the request has been blocking 2 seconds, the request will return instantly. But if the change has already occurred or occurs in less than 2 seconds, the engine will make sure the request blocks at least 2 seconds before returning. The reason for this is that often there is a flurry of changes in rapid succession. For example if a user runs a scene that turns on 20 lights, without the minimumdelay, lu_sdata would be constantly incrementing the dataversion and returning immediately for several seconds while the scene is running, and it would max out the cpu of both the engine and the controller having constant polls.

If the last lu_sdata request failed to return anything, then you must introduce your own delay to prevent from bogging down the controller or engine in a constant polling loop.

On the initial request it is harmless to pass in 0's for the loadtime and dataversion. This is the same as omiting them and means you will receive the full set of data. Also, when the user has made changes to his configuration, like renaming a device, adding a device, scene or room, etc., the loadtime value will change and you will get a full set of data also (full=1). While the engine is running, status changes only, like a light going on and off, etc., will only increment the dataversion. So when you pass in a value for loadtime, if it still matches the current loadtime timestamp of the engine's configuration, then lu_sdata will not return sections, rooms, or categories, and it will only return the scenes and devices that have changed since the dataversion you passed in. It also will not return the name of the scene or device, since changing the name results in a new loadtime.

Thus the back end poll loop may want to set a 'full' variable to 1 whenever it receives a full set of data, in which case the front end user interface knows to re-render the entire UI. When the poll loop receives partial, or diff, data, it can simply update the internal object or variable that stores the device or scene and set a 'dirty' or 'updated' flag for that scene or device, which is how the front end user interface knows to update that scene or device if it's currently shown on the screen. So, when a device is turned on or off, the poll loop's pending lu_sdata will immediately return with the new state of the device, and if that device is shown on the user interface, the UI should immediately change to reflect the new status of the device.

Following is some pseudo-code showing how the poll loop would typically be written:

{{{ int LoadTime=0; int DataVersion=0; int DefaultTimeout=60; int DefaultMinimumDelay=2000; int CurrentMinimumDelay=0; int CurrentSleep=2000; int EngineState=-2; // Meaning we are not connected string IpAddress; // Will be: "http://76.168.224.30:3480/" or maybe "https://fwd2.mios.com/john/tokyo/10266/" int NumFailures=0;

while( Quit==false ) { URL = IpAddress + "data_request?id=lu_sdata&loadtime=" + LoadTime + "&dataversion=" + DataVersion + "&timeout=" + DefaultTimeout + "&minimumdelay=" + CurrentMinimumDelay; string Data = FetchURL(URL);

// If the request was successful, there will be something in Data if( Data.IsEmpty() ) { // Be sure the user knows we're not connected EngineState=-2;

NumFailures = NumFailures + 1; if( NumFailures > MAX_FAILURES ) { CheckConnection(); continue; }

// The request failed, so sleep a couple seconds before trying again Sleep(CurrentSleep);

// No need to introduce a minimum delay since this will be the first request CurrentMinimumDelay=0;

// Try again continue; }

// So we have data. Parse it and update our variables, like the EngineState, LoadTime, DataVersion, and all the scenes, devices, etc. ParseResponse(Data);

// We got valid data, so introduce the minimumdelay in case there's a flood of changes CurrentMinimumDelay=DefaultMinimumDelay; } }}}

The is not complete code, but rather just an outline of the polling process. The assumption here is that there is other initialization code which sets variables, like IpAddress, based on the information from Mode 1/Setup, namely the locating of the engine. while( Quit==false ) means this is a loop that keeps going until the wants to quit the application. In reality when the user chooses to quit you'll need to kill the thread since the FetchURL could be blocking for a while waiting for changes.

On the first pass URL will have loadtime=0 and dataversion=0, meaning we will always get the full list of sections, rooms, etc. So, when ParseResponse parses the json data, it will create all the objects and variables the front end will use to show this to the user.

If the fetching of the URL fails, you should set the engine status back to the initial value of -2, which means the user's UI will be interrupted so he cannot continue, and he'll have a modal 'Please wait... Connecting'. It is normal for the engine to become unavailable for up to 45 seconds or so. For example, if the user just installed some plugins, or saved a bunch of changes, the engine will go offline and stop responding while it's resetting. Therefore, when the fetching of the URL fails the first time you should display the 'Please wait' but don't treat it as an error. That's why we have a if( NumFailures > MAX_FAILURES ), and presumably, since we have a delay of 2 seconds for each request, MAX_FAILURES may be 30 if we want to go to an error condition after 60 seconds of failures. In reality, if FetchURL blocks for a while, having a fixed number of failures, like MAX_FAILURES, may not be good. If FetchURL blocked for 60 seconds, you wouldn't want the user to have a 'Please wait...' for up to 30 minutes. Also, the 'Please wait... Connecting' modal dialog needs to have a 'Quit' and 'Setup' option so the user can exit the app, or go back to the setup part to change his connection. CheckConnection is a function that should take care of a problem by reporting this to the user and letting him check his network settings. In reality, if FetchURL fails, and if you are currently connecting locally, and you are using a mobile phone or other device that can switch from a local connection like wi-fi to a remote connection like 3G, then if you are on a local connection and it suddenly drops, you should right away run a process that checks if the local connection is ok, and if not, switches to the remote connection (ie the https://fwd2.mios.com). Similarly, if the network settings have changed and you were on a remote connection, but now are on wi-fi, you should check if the local connection is ok. This is assuming the user chose the default 'Automatic connection' back on the setup connection page (see: "And also a multiple choice asking the user if he wants: Automatic connection, Local connection only, or Remote connection only."), otherwise you will need to stick to a remote or local connection only.

If the connection fails, you can reset the minimumdelay on the next request to 0, because there's no reason for the engine to wait a couple seconds before return on a lu_sdata request for initial data; you don't want to slow down the user's experience. But, you need to introduce your own sleep of a couple seconds between requests so you're not maxing out the cpu on the engine or the controller.

In this code, it's assumed ParseResponse is the function that parses the JSON data and creates all the objects and variables in memory which the front end will render for the user. If ParseResponse sees the JSON tag "full": 1, then it should purge it's list of devices, scenes, rooms, sections, categories, and rebuild them from scratch, and set a flag so the user interface will re-render the new data. ParseResponse also needs to set the dataversion and loadtime that are passed on the next request to whatever just came in on this request. ParseResponse should also set the variable with the state of the engine (EngineState in this example) to the 'state' tag that comes back in the json data. This way the user interface can show the actual state, or the 'Please wait... Connecting' if the state is -2 (ie the controller is not connected to the engine).

If we got a valid response, then the back end should not introduce its own delay, it should simply set the minimumdelay and let the engine handle the delay (CurrentMinimumDelay=DefaultMinimumDelay). This way, you will not introduce any unnecessary latency in the user interface.

The front end user interface

The engine, the devices and the scenes will all report one of those 4 'states'. In all cases the 'none' state is represented by either gray or an absence of color, and in the lu_sdata request (ex

, all the devices, and all the scenes, can be in one of 4 states:

job_None=-1, // no icon job_WaitingToStart=0, // gray icon job_InProgress=1, // blue icon job_Error=2, // red icon job_Aborted=3, // red icon job_Done=4, // green icon job_WaitingForCallback=5, // blue icon - Special case used in certain derived classes job_Requeue=6 // If the job was aborted and needs to be started, use this special value so we know to clear the m_bQuit flag before starting it

job_InProgress=1, // blue icon job_Error=2, // red icon job_Done=4, // green icon none (nothing is going on with the device or scene), pending (it's actively being controlled), success (the last action performed was successful), error (something is wrong with the device)

You should have a global variable in your app with the engine state.