Developing my weather station, Work in Progress



  • Developing my weather-station

    The aim of this project is to create a weather-station from scratch.

    Several sensors will be integrated. One of them is of an experimental nature. An other one is self-made. The data of some will be compared to see the differences between the different types of sensor measuring the same quantity.

    The mechanical part is not described here.

    Sensor integration, data transmission, data processing and data presentation are addressed.

    Sensors

    The following sensors are applied:

    BME280 : pressure, humidity and temperature
    TSL2561 : light level
    ML8511 : UV level
    DS18B20 : temperature
    HMC5883: compass for the wind direction
    YL38 : rain start detection
    Tipping bucket rain volume sensor.
    Windsensor Rev. P : a hot wire anemometer with temperature sensor

    They will be combined in one single mechanical construction.

    They are hooked-up to one LOLIN D1 mini. This module contains a microcontroller able to transmit via incorporated WiFi capabilities. Several GPIO and communication protocols are available.

    The following sensors use the I2C protocol and need 3.3V power:
    BME280, TSL2561, HMC5883

    The following sensors provide an analog output and need 3.3V power:
    ML8511, YL38

    The following sensor uses the "one wire" protocol and needs 3.3V power:
    DS18B20

    The Windsensor Rev. P has two analog outputs, one for the temperature and one for the windspeed.
    It requires 12V power.

    The tipping bucket works with two simple switching signals of 3.3V.

    All sensors have 3.3V outputs. One requires 12V powers supply, the rest need 3.3V power supply.

    Datacollection

    Per sensor quantity it is adjustable which interval between measurements is applied.
    It might happen that values change suddenly. To be able to catch those rapidly changing values also, per quantity an adjustable difference between sensed values is applied to collect and transmit data. This results in a regular update of data per sensor. On a sudden change in value each sensor transmits extra data.

    Datatransmission

    For data-transmission the MQTT protocol is used. Sensor values are published in each sensor's unique topic.

    Data processing and presentation

    eGeoffrey is used for data processing and presentation.



  • I think that I have enough insight in the way eGeoffrey is working to proceed with my own weather station.

    My intention is to inform you about the progress of the development every now and then.

    At the hardware side I have had some difficulties. They were in the area of creating the PCB for the weather station. This PCB is required to combine the different sensors with the ESP8266 module, a Wemos D1 mini. Because this is my first attempt to integrate a bunch of different sensors with different interfaces I made mistakes in the design of the PCB. But I am getting along.

    Because there are more ways to transmit the data from a sensor with the MQTT protocol to the broker a decision had to be taken which way to go.

    Next to that I have been thinking about how to allocate topics to the sensors. Because the weather station is only the first step in developing my house automation system I developed a scheme for it.

    I would like to share my thoughts about this:

    Sensor network

    Transmission network

    In essence there are two different ways to exchange data between sensors and eGeoffrey using MQTT via the broker, Mosquitto:

    • each sensor module uses its own Wifi link

    • the sensor modules build a meshed network with one of the nodes making one single WiFi connection.

    The mesh network is self establishing. The only user data required to build the network are data related to the WiFi network and the broker the mesh should connect to. Adding and removing nodes from the network is taken care of by the network without user intervention. As long as a module can establish a single link with one of the other modules, it can exchange data with Mosquitto.

    Even at houses with larger plots it is possible to create a sensor network without adding extra hardware to the WiFi network. The sensor modules themselves create the radio network. The module with the best WiFi signal creates the single WiFi connection of the network.

    Sensor control

    Each sensor is controlled by an ESP8266 module. Each module can handle several sensors.
    The data of each sensor have to be published under its unique topic. When dealing with multiple sensors some form of standardization supports maintaining the network.

    Topic allocation

    Each module is on a plot. This plot has a name. This name is the root topic.
    Each module is at a location on the plot. This location has a name. (e.g. living, garden etc) This is the next part of the topic. At one location there can be several modules. Each module has a function (e.g. burglary alarm, light switch). The function name is the following part of the topic. When there are more modules with the same function at one location, the next part of the topic is the number of that module.
    At last we have the type of the sensor itself. One sensor module can generate data of more sensors. If so the sensor module name is the following part of the topic. The last part of the topic is the name of the magnitude the sensor is measuring.

    Examples

    A temperature sensor in one of the two weather stations in the garden of our house would report its data under topic “/ourhouse/garden/weatherstation/2/bme280/temp”.

    A humidity sensor in the living of our holiday home would report its data under topic “/holidayhome/living/general/bme/hum”.

    The topic for the data of the radar sensor of the same module would be “/holidayhome/living/general/radar”.



  • I am integrating eGeoffrey with my weather station. Some early "lessons learned" are already there.

    To investigate what happens with data being processed, it is helpful to filter out certain data.

    Heartbeat data are the first to be filtered out.

    The topic structure used does not allow this.

    The topic structure needs amendment

    The old topic was: "ourhouse/garden/weatherstation/heartbeat

    The new topic is "ourhouse/haertbeat/weatherstation"

    Now the heartbeats of all modules can be filtered out easily.

    Furthermore it is useful to be able to adapt the interval between transmission of sensor data.
    This can be done by sending a signal to a specific module.

    In the software of each module each sensor has a transmission interval for normal operation and one for testing. A variable "testFlag" controls which of both intervals is used.
    The value of this variable is toggled by a message from eGeoffrey.

    Topic /onshuis/test/"module-ID"

    By filtering topics specific data can be examined. Switching data transmission of a sensor on or off is not useful.



  • I have added an extra feature.

    During normal operations the data rate is rather slow for most sensors. For testing purposes it might be handy to have sensor data transmitted at a higher pace.

    This is implemented by a "control an on/off switch" widget. The sensor of this widget sends a "1" or "0" in the topic "/mesh-in/module-ID/ourhouse/test/weatherstation".

    The software of the weather-station receives this message and instead of the operations interval the test interval is used for data transmission. The weather station sends the received message return in the topic "/mesh-out/module-ID/ourhouse/test/weatherstation.

    Egeoffrey has a sensor which receives data in the topic "/mesh-out/+/ourhouse/test/weatherstation".
    A widget "Status – display an on/off status" presents the value of the data reveived from the weather-station.



  • @user2684

    I have a question about data rates of sensors. Where you operate your own weather-station also you might be able to help me.

    Which intervals do you apply for temperature, windspeed, wind direction, light and uv?



  • Gaining experience with the weather station having multiple sensors measuring the same quantity an other idea popped-up.

    Adjusting sensor data level

    Having multiple sensors reading the same quantity it is clear that those sensors deliver data that do not match each other. To be able to compare the results of the sensors measuring the same quantity it is necessary to adjust the readings of each sensor within certain limits. The result is that the curves of the presented data are moved higher or lower on the screen. They can be put over each other.

    Each sensor needs to be adjustable.
    Egeoffrey has the "slider" widget. With the slider a value between limits with a certain step can be sent. It needs to be connected to an eGeoffrey sensor which publishes data to a topic.
    The topic is "mesh-in/module-ID/onshuis/adjust". The module-ID is the number which is used by the mesh network to identify the individual nodes, the chip-id.
    The next topic we need is one to select the sensor to be adjusted. It is "mesh-in/module-ID/onshuis/sensor".

    Each change in value is published. To confirm the settings an acknowledgement has to be given.
    This acknowledgement is published in the topic "/mesh-in/module-ID/onshuis/ack".

    For each sensor we need a widget that displays the adjustment applied to the reading. Per reading an eGeoffrey sensor is needed receiving the adjustment values of the sensors.
    A subscription is needed to the topic "mesh-out/+/onshuis/adjust/sensor-ID".

    Data display

    To gain a better overview of the data presented some grouping of data is needed.
    Per quantity a separate page seems feasible. Data and widgets displaying the individual adjustment of sensors delivering data of the same quantity are visible on one page.



  • Which intervals do you apply for temperature, windspeed, wind direction, light and uv?

    Personally I do not go below the 10 minutes interval just because there are few use cases you need so many data points unless you have critical actions to be executed based on those values and their accuracy. I have also a bunch of zigbee-based xiaomi devices for which the interval cannot be configured and they push new measures every now and then but I'm still fine with it.



  • @user2684
    Would it be possible to enhance the features of the "input widget"?

    At the moment under "allowed values" e.g. ON,OFF" is possible. It would be helpful if a range for numbers could be inserted here. In my case it would prevent inputting wrong numbers for the selection of a sensor controlled by one module.



  • @user2684

    I do have a serious problem with ESP628MQTTMESH. Outgoing everything is OK but incoming the messages are arriving but the callback routine is not triggered anymore. I put an issue in Github, but nowadays reactions take a rather long time.
    So I can not take any control over the weather station via MQTT now.



  • Would it be possible to enhance the features of the "input widget"?

    https://github.com/egeoffrey/egeoffrey-gui/issues/61

    I do have a serious problem with ESP628MQTTMESH.

    This is not related to eGeoffrey right? Just to understand if I can do something or not. Thx



  • @user2684
    The mesh problem has nothing to do with eGeoffrey.
    I am now organising the web page setup.
    I aim to have an overview without having to scroll too much.



  • @user2684

    I fixed the mesh problem. I am now working on controlling the sensors via MQTT by means of eGeoffrey.

    I remark about sensor configuration:
    when "Mqtt service" is chosen and e.g. "actuator" one inserts the topic on tab "configuration".

    When one later on changes the "actuator" to "push" the data on the "configuration" tab are deleted.

    Would be handy if these data would be retained and not deleted.



  • The tsl2561 sensor turns out to be overloaded very fast. That means the reading is 6555 and that is it.

    I ordered a different type the VEML 7700 together with a I2C type UV index sensor the SI1145.



  • Surveying the CPU temperature is a good idea.
    The temperature reading I saw was around 65 degrees centigrade. That could be a little bit high. For that reason I mounted the Raspi in a different casing. The result is worthwhile.

    OtherCasing.jpeg

    The casing of my choice is:

    RPI_CASE_ALU_07F_01.jpg

    I purchased it from "reichelt.de"



  • Using MQTT in two directions, from the sensor module to eGeoffrey and the other way around, turned out to be very simple.

    During development and testing it is handy to control several aspects of each individual sensor.
    The following aspects will be made adjustable from within eGeoffrey:

    • on/off

    • operational data transmission interval

    • test data transmission interval

    • operational/test

    • adjust measurement level

    To send all these parameters from within eGeoffrey a sensor control page is made.
    On this page one widget is used to select the number of the sensor.
    Than five widgets are used to select the aspect or set the value.
    A last widget is used to sent an “acknowledgment” to the module.

    The sensor module receives the MQTT messages which are transmitted by eGeoffrey. As soon as the “ack” is received the data are processed and stored into EEPROM to be used from now on.



  • The idea explained above is working. eGeoffrey sends data to the sensor module and the module sends the requested data in return. That means that with the sensor control page from 12 sensors five aspects can be controlled with eGeoffrey.

    To give you an impression of how that is achieved a little bit more detail is below.

    eGeoffrey sending data

    Topic structure

    onshuis/sensor
    onshuis/status
    onshuis/opsinterval
    onshuis/testinterval
    onshuis/opstest
    onshuis/adjust
    onshuis/ack

    sensors publishing with topic:
    mesh-in/XXXX/onshuis/sensor
    data: integer 0 .. 11

    mesh-in/XXXX/onshuis/status
    data: integer 1 / 0

    mesh-in/XXXX/onshuis/opsinterval
    data: integer 1 .. 60 (minutes)

    mesh-in/XXXX/onshuis/testinterval
    data: integer 1 .. 20 (seconds)
    mesh-in/XXXX/onshuis/opstest
    data: integer 1 / 0

    mesh-in/XXXX/onshuis/adjust
    data: float -5 .. 5

    mesh-in/XXXX/onshuis/ack
    data: integer 1

    widgets:
    onshuis/sensor
    type: input box

    onshuis/status
    type: on/off switch

    onshuis/opsinterval
    type: input box

    onshuis/testinterval
    type: input box

    onshuis/opstest
    type: on/off switch

    onshuis/adjust
    type: slider

    onshuis/ack
    type: on/off switch

    To be sure that the data are received correctly the sensor module sends the data back to eGeoffrey.
    Six widgets are used to display the data received from the module.

    eGeoffrey receiving data

    sensors receiving with topic:
    mesh-out/+/onshuis/sensor
    data: string

    mesh-out/+/onshuis/status
    data: integer 1 / 0

    mesh-out/+/onshuis/opsinterval
    data: integer 1 .. 60 (minutes)

    mesh-out/+/onshuis/testinterval
    data: integer 1 .. 20 (seconds)

    mesh-out/+/onshuis/opstest
    data: integer 1 / 0

    mesh-out/+/onshuis/adjust
    data: float -5 .. 5

    widgets:
    onshuis/sensor
    type: display latest value

    onshuis/status
    type: display on/off status

    onshuis/opsinterval
    type: display latest value

    onshuis/testinterval
    type: display latest value

    onshuis/opstest
    type: display on/off status

    onshuis/adjust
    type: display latest value

    Also it is possible to interrogate a module and retrieve the settings from a sensor.

    eGeoffrey retrieving sensor settings

    Topic structure:

    onshuis/retrieve
    onshuis/ack

    sensors publishing with topic:

    mesh-in/XXXX/onshuis/retrieve
    data: integer 0 .. 11

    mesh-in/XXXX/onshuis/ack
    data: integer 1

    widgets:
    onshuis/retrieve
    type: input box

    onshuis/ack
    type: on/off switch

    eGeoffrey resetting sensor module

    After finishing a test series it is handy to reset the module to the default values with only two actions.

    Topic structure

    onshuis/default
    onshuis/ack

    sensors publishing with topic:

    mesh-in/XXXX/onshuis/default
    data: integer 0 .. 11

    mesh-in/XXXX/onshuis/ack
    data: integer 1

    widgets:
    onshuis/default
    type: input box

    onshuis/ack
    type: on/off switch



  • When one later on changes the "actuator" to "push" the data on the "configuration" tab are deleted.
    Would be handy if these data would be retained and not deleted.

    Understand but when something is saved, I loose previous configuration so bot something easy to implement as for now 😕

    Surveying the CPU temperature is a good idea.

    Very nice chart! 🙂

    The idea explained above is working. eGeoffrey sends data to the sensor module and the module sends the requested data in return. That means that with the sensor control page from 12 sensors five aspects can be controlled with eGeoffrey.

    Good! Just in case you want to share a screenshot of the page you have built once completed, I'd be really curious 🙂



  • Using the same technic it is also possible to turn all sensors on or off and to switch all sensors from test mode into operations mode. Last but not least the possibility to evoke a reboot of a module is added.

    Result

    A sensor module has several sensors.

    Each sensor transmits data with an operational interval. For testing purposes each sensor can transmit data with a test interval. When the quantity measured changes rapidly above a "delta" value extra data is transmitted to catch this kind of rapid strong variations.
    Furthermore the measured value of each sensor can be modified with a positive or negative float value. The aim is to move the curve representing the data of the sensor up and down on the screen.

    The values mentioned above are controlled by eGeoffrey. With eGeoffrey the following aspects of each sensor can be modified:

    • on/off

    • ops/test

    • ops interval

    • test interval

    • delta

    • adjustment value

    When an aspect of a sensor is changed, the value of each aspect is displayed.
    Next to that each sensor can be interrogated to transmit the value of it's aspects.
    All values can be reset to their default values.

    All sensors together of each module can be switched:

    • on/off

    • from ops to test and vice versa

    A module can be rebooted.



  • Some screenshots of the system as it is now:

    1small.jpg

    On this part of the page one can select a sensor and change the values of the aspects. "kies sensor" is used to select a sensor.

    "bevestig bericht" transmits an acknowledgement. On receipt the changes are performed.

    2small.jpg

    "opvragen data" facilitates querying a sensor.
    under "data" the value of all aspects are displayed.

    3small.jpg

    "reset module" enforces resetting the aspects to their default values.
    "totaal" designates the buttons to change settings of all sensors together



  • I like it a lot!


Log in to reply