My general sensor system



  • Overview sensor system

    The sensors are controlled by ESP8266 modules.

    The ESP8266 modules are interconnected with the ESP8266MQTTmesh network.

    The transmission protocol is MQTT.

    The MQTT broker is Mosquitto.

    The data transfer protocol is the "mysensors" protocol.

    Data are processed and displayed by "eGeoffrey"

    Sensor controller

    The software is developed and built in a modular way.

    Parts of the software are derived from my Wireless Track Control System. This system is made to control tracks and sensors of my model railroad. WTCS is operational. It consists of decoders interconnected by the ESP mesh network.

    Per type of sensor the data specific for that sensor are processed in the ESP module to be transmitted using the mysensors protocol over MQTT.

    The mesh only transmits "char" type of data.
    This means that data from a sensor have to be transformed into "char" data type.

    Also the format of the mysensors protocol has to be constructed.

    Data transfer

    Simulated data from a simple on/off sensor and simulated negative temperature data with one decimal are processed and transferred to Mosquitto.

    Mosquitto is running on my LAN-server.

    A separate PC is receiving the test data correctly in the right format.

    Data processing and presentation

    The next step is to install eGeoffrey with the "mysensors" interface on a Raspi 3B+.

    When eGeoffrey is up and running knowledge has to be gathered about the way eGeoffrey has to be tuned to have the data processed and displayed as envisioned.

    Have fun

    Ellard



  • In the following posts I will explain the procedure I used to come to a working system.

    Installing and adapting eGeoffrey

    Starting point is a Raspi 3B+ with the OS on USB-stick instead of on SD card.

    Nowadays not only Raspberry Pi OS (old: Raspbian) is available but also some others of which "Ubuntu MATE" is one.

    Because I do not like some features of Raspberry Pi OS, I decided to give Ubuntu MATE a try.

    I am using Linux Mint on my laptop. That is an Ubuntu flavour. On the laptop I got acquainted with the network manager. Especially this software tool is quit different from the one of the Raspi.

    Installing this OS on Raspi is straight forward. Burn the image onto the USB stick instead of the Raspberry Pi OS and the rest is the same as with the latter.

    In my house I have a network with several WiFi access points connected to the wired LAN.

    During installation of the OS on the Raspi, my network server allocated an IP-address to the Raspi via DNS. I do not provide fixed IP-addresses. The validity (lease-time) of the addresses the server provides is very long.

    On that server an MQTT broker, mosquitto, is also always running.

    Once the Raspi is up and running only every now and than the peripherals are needed.

    Configuration of eGeoffrey is done via webbrowser.

    Because interaction with the Raspi is required only for installation of the core of egeoffrey and some extra modules, I think it is not worthwhile to spend time on some remote access possibilities like VNC for this. Just use the monitor, keyboard and mouse hooked-up to the Raspi directly.

    OK lets get started with egeoffrey.

    Open a terminal window on the Raspi.

    As far as I experienced the commands require "root" rights to be performed. So use "sudo" to issue the commands.

    Issue the command:

    sudo bash -c "$(curl -ssl https://get.egeoffrey.com)"
    

    Now lean back and wait until several software packages have been downloaded and installed.

    To see if everything is OK in the terminal window goto the "egeoffrey directory"; issue the command:

    cd egeoffrey
    

    Now issue the command:

    sudo egeoffrey-cli status
    

    It takes some time but than you have an overview of the status of the different software modules of egeoffrey.

    The status of each component should read "Up".

    To see if the website of egeoffrey is also up and running, on a separate machine log-in in the network of which the Raspi is a user. Point a browser to the IP-address of the Raspi by inserting the IP-address in the URL field of the browser.

    It might take a little bit longer than you are used to but the welcome page of the egeoffrey website should be presented. Standard you are logged-in as "guest".

    If you see "bells and whistles" egeoffrey is OK.

    Now we have to proceed to extend the standard eGeoffrey with the "mysensors" module.

    Go back again to the Raspi and in the terminal window in the "egeoffrey directory" issue the command:

    sudo egeoffrey-cli install egeoffrey-service-mysensors
    

    than issue the command:

    sudo egeoffrey-cli start
    

    Now the actual download and install procedure starts.

    Lean back again until everything is ready.

    To see if everything is OK just request the status of eGeoffrey again. Issue the command:

    sudo egeoffrey-cli status
    

    Again everything should be "Up".

    This concludes the installation of eGeoffrey. If no surprises pop-up it is a piece of cake!



  • Now we are going a step further.

    Now on a second machine, point your webbrowser to the IP address of the Raspi on which you installed eGeoffrey.

    You are logged in as "Guest".

    To get to the Admin part of the website you have to log out and log in as Administrator.
    The username is "admin" and the password is "admin".

    Log out by clicking on the "arrow" before "Guest" in the left panel.

    In the panel in the middle under "User" replace "Guest"with "admin". Insert "admin" in the field "Password".

    Check what is there:
    Left pane – ADMINISTRATION
    Three lines:
    House
    eGeoffrey
    Help

    House:
    Settings
    Sensors
    Rules

    House leads to "House configuration" window.
    In all descriptions a "*" is contained. All fields need to be filled.

    Name of your house: insert what you like.

    Timezone offset: google on "timezone offset". Via the "Wiki" article you can find the offset you need. Insert the number into the field.

    Language: only "en" is possible.

    Units: select "Metric" or "Imperial".

    Latitude: For the location of your house, click the link. If everything went OK, your location is given in two ways: DD (decimal degrees) and DMD (degrees, minutes, seconds)

    Take the DD values with two decimals. Insert the Latitude value into the field Latitude and the Longitude value into the field Latitude.

    Confirm the data by clicking the "Save" at the bottom right side of the window.

    The next tab on the "Settings" page is "Web interface". Leave it for the time being.

    The next tab on the "Settings" page is "Users".

    Leave the "admin" data untouched for the time being. The same pertains to the data of "Guest"

    The possibility to add users is activated by clicking on "guest".

    Now on the bottom right side of the panel the possibility to add, delete and save a user becomes available.

    Click on "New User". A small window pops up. Insert the username of the user.
    The new user is now available. Next the surname has to be inserted directly into the field.

    Than the field "Icon*" needs to be filled (with what?). I inserted the username.

    After inserting a password into the "Password" field, save the data with the "Save" button.

    The next tab is "Groups"

    Three groups are available:
    egeoffrey_admins with user "admin"
    guests with user guest
    house_admins with user admin

    At the bottom right side four buttons are available:
    New User, New Group, Delete Group and Save.

    The use of "New User" is not clear.
    Clicking the button presents a new field to insert the name of a new user of the selected group.

    To add a user to a specific group select that group in the most left column.
    Than click "New user". A new line shows up. A "red cross" is next to the filed. Clicking on the cross deletes the content of the field.

    Clicking "New Group" provides a pop-up window to insert the name of the new group.
    Users can be inserted into the new group.

    The use of creating a new group is not clear.

    For the issues which are not clear I will post a separate question.



  • Thanks again for keeping track of your progress this helps me in understanding typical user's process and I will for sure take inspiration from this for enhancing the current documentation!



  • Some weird things happening having nothing to do with Geoffrey but with Raspi OS.

    I gave Raspi OS in the most recent version another try.
    The Wifi manager is now OK.
    When I do NOT update/upgrade the OS and go then to eGeoffrey market place, the modules are listed.

    After update/upgrade the marketplace is not accessible anymore?????????

    Going to try some things to look if I can find a workaround.



  • @eporocrail wonder if the update includes some new curl/wget version which requires different switches than the one I am using. Would you mind detailing the steps so I can fully reproduce? (e.g. which raspbian should I start with, which upgrades, etc.) Thanks!



  • @user2684

    I am using RaspioOs
    Version: 2020-05-27-raspios-buster.armhf.img
    Release 3.3 May 2020

    Install without automatic update.
    Access to martketplace is OK

    Manual update some info from terminal:

    Unpacking rpi-eeprom (7.2-1) over (7.0-1) ...
    Preparing to unpack .../20-raspberrypi-ui-mods_1.20200603_all.deb ...
    Unpacking raspberrypi-ui-mods (1.20200603) over (1.20200514) ...
    Setting up rp-prefapps (0.22) ...
    Setting up rpi-chromium-mods (20200602) ...
    Setting up raspberrypi-kernel (1.20200601-1) ...

    After this update I have no access any more to the marketplace



  • @eporocrail ok so if I understand correctly you start from 2020-05-27-raspios-buster.armhf.img then apt-get update & apt-get upgrade and when trying to install a new eGeoffrey package you get the behaviour you described on https://forum.egeoffrey.com/topic/15/installing-new-service/3, correct? If so I'll try to reproduce the problem. Thanks



  • @user2684
    Before the update I can access the marketplace with web browser.
    After update/upgrade manually I can not access the marketplace anymore.
    The webpage keeps displaying "loading" and nothing happens.



  • @eporocrail ok so the problem is when accessing with the web browser, not with the CLI, my bad in understanding. Are you using the browser on the raspi or on a different device to access? If you experience the problem with both my guess is that the thing has nothing to do with the upgrade but when you refresh multiple times the marketplace website, since it is building the page on the fly by pulling information from github, when reloaded multiple times, github is blocking too many requests (try visiting https://api.github.com/repos/egeoffrey/egeoffrey-marketplace/contents/marketplace for confirmation). If so I need to add an explicit warning or something for this. Thanks



  • @user2684 I think you are right. I was able to install the mysensors module.
    But that would mean that Github is blocked VERY fast.

    I expect to proceed with the installation not earlier than in two days.



  • @eporocrail looks like the case, I had a similar issue a while ago but with many many times the page realoaded, but I see this happening faster now, could easily be github lowering down the limit. I've opened up an issue to fix this (both for the marketplace and the CLI - https://github.com/egeoffrey/marketplace.egeoffrey.com/issues/1), it could take a while since I need to figure out the best way to keep a local cache, hope it is not a blocking issue for now.



  • @user2684

    I did a complete new install starting with Raspi OS.
    I was able to install Geoffrey and the mysensor service. No problem with GitHub.
    This time I was not logged in as "Guest" but as "admin".

    Inserted the details of the house.

    Started inserting a new sensor.
    I filled the fields of the first tab. I switched to "Advanced editor" and back again.
    The content of the fields I just inserted, was gone.

    I inserted the data again and clicked "Save". The system did not only "save" but also "close". After a "save" one leaves the sensor configuration.

    Where do I find information about the values for the field "Sensor Icon"?

    On the tab "data" it is not clear what "format" means. I wanted to insert a simple on/off button. What "format" do I have to select?

    Where I am using MQTT I would expect to insert data for the broker somewhere.
    Under "Sensor Configuration" there is no place for these data.

    I found it under "eGeoffrey - modules"

    Later I will continue configuration and testing
    I will report in a separate post



  • @user2684
    Some more experiences of which some require your attention.

    Register a new sensor.

    Weird behaviour: as soon as the the data on tab "General" is saved the input window is closed instead of data only being saved.

    After inserting data on tab "general data" go to "Advanced Editor". All empty. Go "Back".
    Now the newly inserted data are gone.

    Unclear what data is inserted into filed "Sensor Icon".

    Do NOT save.

    Tab "Data"
    Field "Format" is not clear. I want to insert a push button as sensor. I would expect to insert something like "On/Off".

    Tab "Processing"
    Automatic aggregation. Select from pull-down menu "None".
    Retention Policies. "single_value".
    Convert acquired value before being saved. "1_to_ON".
    Convert input value before being sent to an actuator. "None".

    Tab "Service"
    Associated Service. Select from pull-down menu "mysensors_mqtt: Interactwith a Mysensors MQTT gateway"
    How to interact with the gateway. "Push – the sensor will periodically provide new data".

    Tab "Configuration"
    The unique identifier of the node. "11".
    Each node can have several sensors attached. This is the child-id that uniquely identifies one attached sensor. "5".
    Mysensors command to trigger. Select from pull-down menu "SET".
    Mysensors type. "S_BINARY".

    Configuring the "mysensors" module.

    Under "eGeoffrey" goto "Modules".

    At the bottom three versions of the "service/mysensors" are visible and running.
    For the "Ethernet" and "Serial" version click on the "arrow-down" in the "Actions" field.
    Click on "Stop module".
    Click on the "arrow-down" in the "Actions" field.
    Select "Edit Configuration". Module Configuration check the box "Disable the module".

    Now only the module "service/mysensors_mqtt" is running.

    For the "mqtt" version click on the "arrow-down" in the "Actions" field.
    Select "Edit Configuration"

    Module Configuration.

    The MQTT hostname to connect to. Insert the "hostname" or the IP-Address. "192.168.2.1" in my case.

    The port of the MQTT Broker. Is set to "1883". Leave as is.

    The username for authenticating against the MQTT broker. In my case leave empty.

    The password for authenticating against the MQTT broker. In my case leave empty.

    Topic prefix to subscribe to. "kamer/uit".

    Topic prefix to publish to. "kamer/in".

    Under "eGeoffrey" goto "Icons". Here they are. It is a list with general purpose icons. Not many I think useful for eGeoffrey.

    Looking into the logs it appeared that errors where caused by the mysensor modules which are not used.
    Reboot the Raspi.
    After reboot under "eGeoffrey" "modules" all three mysensor modules are green (running).
    Checked to see if the ethernet and serial version still where checked "Disable module". Yes.

    Stopped both modules. No effect. Both modules keep being interrogated.



  • The marketplace issues has been solved, both for https://marketplace.egeoffrey.com and when using the CLI (please upgrade to v1.0-26 with sudo egeoffrey-cli upgrade). The marketplace website now builds periodically a cache of all the items of the marketplace and both the website and the CLI get information from this cache so avoiding hitting github directly and so be blocked for the too many requests 🙂



  • Thanks once again for these precious feedbacks. Let me reply point by point:

    Weird behaviour: as soon as the the data on tab "General" is saved the input window is closed instead of data only being saved.

    This is expected, the button save and close the window. Do you think a different label on the button would work better?

    After inserting data on tab "general data" go to "Advanced Editor". All empty. Go "Back".
    Now the newly inserted data are gone.

    Still expected (or better a different behaviour would be really difficult to implement). I'd recommend avoiding using the Advanced Editor at the beginning since the configuration schema is not documented yet and there are no options by the wizard. I could take the link away if this is creating confusion.

    Unclear what data is inserted into filed "Sensor Icon".

    This has to be documented better for sure. This field (any icon-related field) is expecting the name of an icon taken from the "eGeoffrey" / "Icons" menu under Administration

    Field "Format" is not clear. I want to insert a push button as sensor. I would expect to insert something like "On/Off".

    I've added a bunch of new pages on the docs website, have a look at https://docs.egeoffrey.com/configure/sensors/ if not done yet, wonder if clear enough. Format is the format of the data supposed to be stored in this sensor (being a sensor a logical container of one or more values). For a on/off sensor you can use "integer" and store 0 or 1. If the remote sensor is instead sending or expecting the string on/off, then would be string.

    Convert acquired value before being saved. "1_to_ON".

    This entirely depends on the logic of the attached sensor. If you set integer and store a 0 or 1 and the remote relay expect 0 or 1 to trigger this is not required. When you set "1_to_ON" means every time a new value comes in (from the remote sensor or from the web interface, the 1 is converted to the string "ON" which is not necessarily what you want to use. The web interface has a on/off widget which expects a 0/1 stored in the sensor.

    Configuring the "mysensors" module.
    Under "eGeoffrey" goto "Modules".

    Yes, this is usually to be done before configuring the sensor (I tried to describe the workflow in this new page: https://docs.egeoffrey.com/configure/workflow/). Configuring the module set up the integration, the sensor leverages then the integration

    Select "Edit Configuration". Module Configuration check the box "Disable the module".
    Now only the module "service/mysensors_mqtt" is running.

    An alternative (more advanced) would be to edit the docker-compose.yml file in the eGeoffrey directory, identify the entry for the MySensors package and under EGEOFFREY_MODULES, removing those you would not need.

    After reboot under "eGeoffrey" "modules" all three mysensor modules are green (running).
    Checked to see if the ethernet and serial version still where checked "Disable module". Yes.
    Stopped both modules. No effect. Both modules keep being interrogated.

    Ok so you're saying even after disabling those modules they are still running after a reboot. I'll try to reproduce it. What about the mqtt one, is everything working there?
    Thanks!



  • @user2684 said in My general sensor system:

    Weird behaviour: as soon as the the data on tab "General" is saved the input window is closed instead of data only being saved.

    I would like to come back to this issue once again. Like in several other windows I would like to only save data when clicking the "Save" button. And I would like to close the window when I click the "Close" button. At this moment when I click the "Save" button data are saved AND the window is closed. The latter action should not be invoked by clicking only the "Save" button.

    After inserting data on tab "general data" go to "Advanced Editor". All empty. Go "Back".
    Now the newly inserted data are gone.

    I think it is a little annoying that by going to a window one step deeper down the road the data inserted into the previous window are deleted.

    "Ok so you're saying even after disabling those modules they are still running after a reboot. I'll try to reproduce it. What about the mqtt one, is everything working there?"

    Yes. Whatever I try to disable the two of the MySensors modules which I do not need, is not working. So from the user perspective all three MySensors modules are running where two of the three modules keep producing a huge amount of errors in the logs.

    I assume that I am able to disable them in the docker.compose.yml file but that is I think not the way to have it done by the less software minded user.

    The second part of your remark is for me not clear.(What about the mqtt one, is everything working there?)
    Do you mean that next to the "MySensors" modules I also have to install extra the "MQTT" module?
    I will give it a try but if this where the case then again this is not clear for the user.
    Maybe it would be more elegant to treat The MQTT module more or less as a "dependency" of the "MySensors" module and have the "MQTT" module installed automatically. (if possible, I am not familiar with docker at all)

    I am going to look into the new documentation and I will try to get some message into eGeoffrey one way or another I can think of.



  • @user2684

    I tried to have a look at the documentation. Something is wrong with the menu structure.

    When I follow the link you provide in your last post I am able to access the documentation. But as soon as I activate the link "Welcome" the access to documents is not working anymore.

    The only way now to get to any useful info is following the above mentioned link. The normal way to access the documentation is disrupted.



  • @eporocrail said in My general sensor system:

    I tried to have a look at the documentation. Something is wrong with the menu structure.

    This has to do with the browser cache I think, try a shift+F5 in the main page, you should see the menu populated correctly. Thanks



  • @user2684

    First of all you did a great job on the documentation.
    Very much has been added.
    I assume you will go through the text again to get rid of the few typos which are still there.

    The last three links (Configure rules, Upgrades and User management
    are leading to "404".
    I think that the links are not OK or just hide the links until the pages are present in the documentation.

    A different question I have is:
    How do I test that a message is entering eGeoffrey via the "MySensors" interface?

    The situation is as follows:
    I push a button. I can seen that my system sends the message in the right format to Mosquitto. The next step is check on a second machine with the terminal if there it is possible to receive that message.
    Thereafter I would like to see that that message arrives in eGeoffrey.

    To put it differently: I want to be sure that the message evoked by the button action is following the path it is supposed to do, step by step.

    By the way, after I managed to get the system with a simple push button up and running I am going to make e step-by-step description for my own documentation. That document I intend to publish on this forum also.


Log in to reply