Data Provider Catalog Plugin

Latest revision as of 02:27, 15 June 2013

This page is OBSOLETE

This has a general overview of the request architecture. For a list of requests see Data Provider Catalog Plugin Requests.

[edit] Introduction

Our goal is to allow developers to expand the functionality of Vera, adding custom UI's, and so on, without any knowledge of C/C++, or any of the under-the-hood LinuxMCE internals, like DCE, DCERouter, etc. If you want to know more about the underlying LinuxMCE DCE architecture, and DCERouter, there is a good introduction here: guide provided by LinuxMCE. The new LuaUPnP API allows you to communicate with all the devices in the system using standard UPnP calls, and you can write your own custom drivers in Lua, instead of using C/C++ and the DCE libraries.

The Data Provider Catalog Plugin provides an easy way to get data, such as device listing, device status, and so on, using standard HTTP calls. You can also use the same HTTP calls to send messages, initiate UPnP actions, and so on.

[edit] Information

The Data_Provider_Catalog_Plugin runs as a plugin for the DCERouter. It acts as an aggregator for data sources. Other devices in the network can register with it by sending the command CMD_Register_Data_Provider and register to provide data, which is identified by a unique ID. This is intended to replace the DataGrid concept previously in LinuxMCE.

So, for example, the media plugin can register to provide a set of data with the ID "SongsByArtist", and a security module can provide a set of data with the ID "ListOfSecurityBreaches". These devices register with Data_Provider_Catalog_Plugin, which then becomes a single point for requesting data.

Request data from Data_Provider_Catalog_Plugin by sending it the message CMD_Request_Data_Provider (ID 971), which includes parameters where you can indicate what format you want the data in (ie XML, plain text, etc.), what input parameters are provided to the data source (ie "SongsByArtist" may require an input parameter of "Artist ID").

The Data_Provider_Catalog_Plugin is in the category 174 (DEVICECATEGORY_Data_Provider_Catalog_Plugins_CONST). One device that registers to provide general purpose data sources is Shared_JSON_Data_Provider_Plug, and one of the data ID's it registers is "alldata", which returns all the common user data in the system (i.e. list of devices, scenes, etc.). It provides it in 'json' format.

So to get this data you can use the MessageSend utility as follows:
/usr/bin/MessageSend dcerouter -o -targetType category 0 174 1 971 10 alldata

NOTE: alldata returns the user_data that was last written to the disk. This may not be up to date if there are local changes. For an up to date list, use the user_data request instead.

The Data_Provider_Catalog_Plugin also includes a small http service which can be used to request data as well as to send messages which listens on port 3451. Previously when outside applications, like a php page, wanted to send messages they would do a system call to spawn the /usr/bin/MessageSend utility. This is inefficient because it makes a system call each time.

To request data, request "data_request" from the HTTP server and include the parameters:

id=id of the data request
parameters=parameters to send on the data request
input_format=format of the parameters being sent
output_format= the desired output format of the data

For example, the following URL returns the same list of 'alldata' described earlier:

http://192.168.80.1:3451/data_request?id=alldata&output_format=json

You can also use this to send messages by requesting "messagesend" and adding the parameters:

from=the originating device ID
to=the destination device id
type=the type of message
id=the message id
response=[r|o|c] for Return String, Output Parameters, and Confirmation of delivery. Default=don't wait for response
target=[c|t] meaning the 'to' id is not a device ID but rather a Category or Template of device. Default=to is a device id
[device data]=some other message parameter where device data is the number

For example, to request a video frame from a camera with the device id #20 you send the message type 1 (command) id 84 (get video frame) and optionally the parameter 60 and 61 (width and height of desired image). The command returns output parameters 19 and 20 where 19 contains the video image and 20 the format. So we want to send message with response=o because we want the request to block until the message is process by device 20 and we want to get back the output parameters. Here is the URL:

http://192.168.80.1:3451/messagesend?from=1&to=20&response=o&type=1&id=84&60=640&61=480

The response will be "OK" if the message succeeds, or "FAIL" if it didn't, followed by the values of the output parameters. Because output parameter #20 is defined as 'binary', the result is uuencoded and the U after the 20 indicates this. The above returns:

OK

[19=3]

jpg

[20U=18373]

[uu encoded video frame]

So parameter 19 is 3 bytes long ("jpg"), and 20 is UU encoded with 18373 characters.

If you are sending commands to a device that creates 'jobs', like for example, sending an On or Off to the Z-Wave module, you should request a response, either response=r or response=o, and the OK will be followed by the job id. Then, if you want to get the status of the job, send a message type MESSAGETYPE_PENDING_TASKS (14) and the ID is the job # to the parent device that handles the job. For example, if you have a device 15, which is a light switch, and it's controlled by device 9, which is a Z-Wave module:

Send the off: http://192.168.80.1:3451/messagesend?from=1&to=15&response=r&type=1&id=193
You will get back something like: OK 2037, so 2037 is the job id
Send the request: http://192.168.80.1:3451/messagesend?from=1&to=9&response=o&type=14&id=2037
and you will get back the status of 2037.

Other plugins can register to provide certain types of data. For example, for each IP camera it manages Generic_IP_Camera_Manager registers the id "camera_control/device #". The parameters it recognizes are: get_video_frame, left, right, up, down, zoom_in, zoom_out

So the following makes the camera with device id #20 move right:

http://192.168.80.1:3451/data_request?id=camera_control/20&parameters=right

and the following returns a single video frame from the camera

http://192.168.80.1:3451/data_request?id=camera_control/20&parameters=get_video_frame

To execute a Scene (aka CommandGroup), send a command 370 (Execute Command Group) with the parameter 28 (PK_CommandGroup) set to the scene number, which you got from the user_data request. For example, scene_5 in the json is scene id 5. Send it to the 'general info plugin', which is a device with the category 61 (DEVICECATEGORY_General_Info_Plugins_CONST), and get a return response:

http://192.168.80.1:3451/messagesend?from=1&to=61&target=c&response=r&type=1&id=370&28=5

To dim device #19 to 50%, send it a dim command (COMMAND_Set_Level_CONST=184) with level 50 (COMMANDPARAMETER_Level_CONST 76):

http://192.168.80.1:3451/messagesend?from=1&to=19&response=r&type=1&id=184&76=50

The return response will be, for example, OK 6467 where 6467 is the id of the job for this particular dim command. The zwave_status request will let you monitor the progress of that job and see if it went through or not.

Try:
http://[ip]:3451/data_request?id=zwave_status&output_format=xml (or output_format=json)
or leave the parameter off for plain text. Use Firefox for XML since it has a built-in XML viewer.

You can also send command 193 (COMMAND_Generic_Off_CONST) or 192 (COMMAND_Generic_On_CONST). For thermostats, send Off, or 280 (COMMAND_Set_HeatCool_CONST) with parameter 8 (COMMANDPARAMETER_OnOff_CONST) set to the letter A, C, or H (auto, cool, heat), or send command 279 (COMMAND_Set_Fan_CONST) with parameter 8 to '0' for fan on auto mode or '1' for fan always on, or send command 278 (COMMAND_Set_Temperature_CONST) with parameter 5 (COMMANDPARAMETER_Value_To_Assign_CONST) set to the set point temperature in degrees Celsius.

If you turn on the FindVera.com remote access, you can also securely access your Vera's port 3451 remotely to control it, fetch devices, etc. with the following syntax:

https://[ra server]/[username]/[password]/3451/

Different installations use different remote access servers. The ra server for a given installation is defined in your system file: /etc/cmh/RA_Server. If you have an application that prompts for a user's username/password, such as a cell phone app, call this to get the ra server for a given username: https://findvera.com/get_server.php?username=[username]

So, with a username of testvera9 and password of myvera123, the following:

https://ra1.findvera.com/testvera9/myvera123/3451/data_request?id=simple_device_list&output_format=txt
is the list of devices
https://ra1.findvera.com/testvera9/myvera123/3451/messagesend?from=1&to=19&response=r&type=1&id=193
turns off device 19

You can remotely access port 80 like this:
https://ra1.findvera.com/host-testvera2/cgi-bin/cmh/get_json.sh?file=template_data.json

[edit] Understanding the messages

The above assumes you understand the basic concept of LinuxMCE messages. The LinuxMCE site contains lots of documentation on messages, and there are user-friendly front-ends for navigating commands, command parameters, etc. The Vera system only uses a small subset of the full LinuxMCE commands. If you're not already familiar with LinuxMCE messaging, here's a quick primer.

Devices have a given "DeviceTemplate", which is the definition for that type of device. For example, "John's floor lamp" may be device #18, and have the device template #38, "Dimmable light". DeviceTemplates will implement certain commands, like a dimmable light implements command #192 "On". Commands take parameters. For example, the "Dim" command may take a parameter whose value represents the dim level. DeviceTemplates also fire certain events. For example, a "motion sensor" may fire "Sensor Tripped". Events also have parameters, such as for "Sensor Tripped", there's a flag "Tripped" which can be 1 or 0 depending on whether the sensor is detecting motion, or stopped sensing motion. DeviceTemplates also reference Data, or configuration parameters. For example, a "Dimmable Light" has a configuration parameter #12 "Port/Channel", which, for a Z-Wave device is the node ID.

In LinuxMCE there are web pages to visually present all this data, and the data is stored in a MySQL database. As an embedded system, Vera doesn't use a database, rather it stores all the data in .json files.

Note that you can also retrieve the raw json files directly using a cgi-bin:
http://[ip address]/cgi-bin/cmh/get_json.sh?file=xx.json
where xx is:
user_data = user specific data, like scenes, timers, events and devices
template_data = static data defining the structure of commands, events, device templates, etc
timezone = the known time zones

Calling http://[ip address]/cgi-bin/cmh/get_json.sh?file=template_data.json will return the raw json data. Note that you can request the user_data.json file, but the data may be out of date. This data may change dynamically as the system is in use and, since the flash memory in the chip has a limited number of re-writes, rather than re-writing the user_data.json file everytime something changes, DCERouter will update an internal copy, and only periodically flush the changes to the RAM drive with user_data.json on it. So for the most up to date user_data, use the request:
http://[ip address]:3451/user_data?output_format=json
or, if you want it in XML format:
http://[ip address]:3451/user_data?output_format=xml

The data file template_data has the master definition for Device Templates, commands, events, and data configuration.

Note that the devices have a hierarchy defined by the FK_Device_ControlledVia. Device 1 is the access point itself, the home. The child DCERouter, in this case device 2, is the message router. The user never needs to see the DCERouter or any of its children presented on the device list because these are used internally.

Some devices are what we call "Interface devices", meaning they represent the interface that handles child devices. For example, device #9 is 'Z-Wave' and device #10 is 'Generic IP Camera Manager'. What the user cares about are the children of device #9 (the Z-Wave devices) and children of device #10 (the cameras). But the interface devices themselves likely wouldn't appear on the device tree you show the users. Note this on the device page.

Everything you do in the system is based on sending messages to devices. A message has a type. The types are:

     enum eMessageType 
     { MESSAGETYPE_COMMAND=1, MESSAGETYPE_EVENT=2, MESSAGETYPE_DATAPARM_CHANGE=3, MESSAGETYPE_REPLY=4,
       MESSAGETYPE_DATAPARM_REQUEST=5, MESSAGETYPE_LOG=6, MESSAGETYPE_SYSCOMMAND=7,
       MESSAGETYPE_REGISTER_INTERCEPTOR=8, MESSAGETYPE_MESSAGE_INTERCEPTED=9, MESSAGETYPE_EXEC_COMMAND_GROUP=10,
       MESSAGETYPE_START_PING=11, MESSAGETYPE_STOP_PING=12, MESSAGETYPE_PURGE_INTERCEPTORS=13,
       MESSAGETYPE_PENDING_TASKS=14 };

You generally just use message type #1, command. Messages have an ID, which is based on the message type. So, for a command (message type 1), id 192 is "Generic On".

You can get more info on the messages by looking in the template_data:
http://[ip]/cgi-bin/cmh/get_json.sh?file=template_data.json

So, when you're looking at the devices, you'll note that the camera has device template #47. Look in template_data for the "DeviceTemplate" section and you'll see #47 is an "IP Camera".

To see what commands you can send an IP Camera, look in: DeviceTemplate_DeviceCommandGroup and see that device template 47, the IP camera, implements the DeviceCommandGroups: 26, 54, and 68 - for example:

"DeviceTemplate_DeviceCommandGroup_47_26": {
"FK_DeviceTemplate":"47",
"FK_DeviceCommandGroup":"26"
},

Now, looking in DeviceCommandGroup_Command we see command group #26 includes command #84 and 973:

"DeviceCommandGroup_Command_26_84": {
"FK_DeviceCommandGroup":"26",
"FK_Command":"84",
"Description":"Gets a picture from a specified surveilance camera"
},
"DeviceCommandGroup_Command_26_973": {
"FK_DeviceCommandGroup":"26",
"FK_Command":"973",
"Description":"Take a picture and store it in the archive"
},

To see more on those commands, look in the Command section:

"PK_Command_84": {
"Description":"Get Video Frame",
"FK_CommandCategory":"9"
},

So the command is called "Get Video Frame" and the CommandCategory is 9. We can see that from command #84 that it takes a couple parameters by looking in Command_CommandParameter:

"Command_CommandParameter_84_19": {
"FK_Command":"84",
"FK_CommandParameter":"19",
"Description":"The video frame",
"IsOut":"1"
},
"Command_CommandParameter_84_61": {
"FK_Command":"84",
"FK_CommandParameter":"61",
"Description":"Frame height",
"IsOut":"0"
},

Parameter #19 is an output parameter and contains the video frame, parameter #61 is an input parameter and contains the frame height, and there are other parameters too.

According to the section "CommandParameter", the description of the parameter #61 is Height:

"PK_CommandParameter_61": {
"Description":"Height",
"FK_ParameterType":"2"
},

and the type of paramter is an int:

"PK_ParameterType_2": {
"Description":"int"
},

Now, look back at the above example for retrieving a video frame:
http://192.168.80.1:3451/messagesend?from=1&to=20&response=o&type=1&id=84&60=640&61=480

You're sending a message of type 1 (a command) and id 84 (get video frame) from device #1 to device #20 (assuming #20 is an ip camera). We want to wait to get the output parameters (response=o), and the parameter 61 (height) is 480.

Now the response, which includes [20U=18373], means parameter #19/#20 contain the image and type of image being returned.