I think this might have been covered at some point on the old forum, but since it’s now seemingly deleted, I’ll ask here. I submitted an issue on github in the hopes that Gene still reviews them.
Basically, if you have a sensor module with a value (temperature for instance) and it doesn’t change significantly over time, the display in the statistics page will show only the last time the value was updated and a missing line up to the current time. When the value does update, it jumps to the new value as though it jumped. What should be displayed is a horizontal line and then a near vertical (potentially) line to the new value. In the case of indoor humidity and temperature, the values change rarely and not significantly. But, that doesn’t mean the display should be blank for days (potentially).
I believe this is intentional to save database size. I believe that Gene coded it so that the database only keeps the values when they change rather than a long string of the same value. This may make sense some times, but I believe this should be an option left to the user. I can provide pictures of the results of this “bug” if it’s not clear.
My question is, does anyone know where this can be adjusted in code. I have looked through the source on github and so far have not located what needs to be changed.
About your question - actually, I also can’t find the place where statistic values are filtered out. It seems that all of them should be in the statistics database. But while investigating this issue I found out that my stats DB is corrupted, because I had manually deleted it and HG had not recreated its inner structure (tables, etc).
But also I can’t reproduce your issue. Here is the screenshot of my statistics page with humidity values. As you can see it has no gaps.
Maybe it’s a version issue? I didn’t see anything relating to the statistics database or display page in the notes so I’m not sure how that would affect things. I don’t display with the bars, so maybe it’s a visualization thing.
I just saved a couple screen shots and maybe it’s the data that’s the issue. I’m not sure. The temperature and humidity are being captured by 3 DHT11 modules connected to and ESP8266 which transmits via MQTT and HG captures. If a humidity exists then a paired temperature must exist. I had an example the other day where the humidity was being updated regularly but the temperature was missing. I should have saved a picture since now it doesn’t show that.
BTW, I’d share the pictures, but I’m not seeing a picture button on this forum.
The following data was for a 1 day period as captured by 3 DHT11 thermo-hygrometers hooked up to an ESP8266. The data was transmitted through my Mosquitto server on the RPi where HG lives. HG saw the data and recorded it into the database. I don’t know what the difference between the database and the displayed graphs are, though I suppose that would be the next step.
As you can see, the humidity has data all the way across for DHT3 (GunSafe3) whereas the temperature has large gaps. Note that in order for there to be humidity, there MUST be temperature based on how the code is written and the data transmitted.
The only difference I can think of is that humidity varies due to both slight changes (maybe) and the fact that the DHT11 is a pretty poor quality humidity sensor so the data varies (definitely). The temperature is fairly stable and I only see rare differences. Also, the temperature only changes by 1°C and the temperature should not change at all really.
I know no one here can fix the issue since Gene coded the database, but it certainly looks like my observations are true. The MQTT log file shows that temperature and humidity are both seen every time data comes in but the database (as seen via SQLlite) shows that temperature is missing for many instances of humidity. I have no idea why and no idea how to further troubleshoot.
It doesn’t sound that hard to fix. Temperature only written to the DB if the value have changed.
If we can find the code it should only be a if-statement to comment out.
Probably a little harder to fix on the graph side. But disk space shouldn’t be that critical. So a few more values in the database shouldn’t matter. So the first solution would be the easiest.
I would like to learn more of the HG code so I will search and see if I can find anything.
If I could locate the code where data was written to the database it should be pretty obvious what the issue is. My guess is either 2 values can’t be written that quickly (thus 1 is sometimes ignored) or there is code that ignores duplicates.
I think the best way to investigate this issue - debug the application and see what actually happens there.
Or write extensive logging around these functions.
The simplest first test I’d to change my MQTT message so the order is reversed. If humidity then becomes an issue, it’s timing. If it doesn’t change anything, it’s either temperature specific logging or repeated value related.
Timing feels very unlikely. If I understand things right, modules write values to a list that is dumped to the database every 5 minutes.
In that case it would be very unlikely that it is either not possible to write the value to this list, or that just the temperature values is missed when the list is dumped to the database.
I’m no expert so I can be wrong, but I don’t really understand that approach. Computers today, even a Raspberry Pi should be able to handle more load than that?
Anyway, the only thing I could find that looks suspicious is line 153, in ValueStatistics.cs. Some compare between the current and previous value. Also strange naming, where Current is the previous value??
FWIW, I had several other links that didn’t show up because the forum decided to add a thumbnail and that made the post too long. I am looking for them again, but I searched the github repository for “statistics”. There are 4 pages of references, but anything with html is only for display so I ignored it. There were around 5 or 6 total files that looked potentially useful.
I don’t have a live system yet with real temp and hum values. Only have the demo environment. But without to much debugging I would just try and comment that if-statement I mentioned above and see if there is a difference. If you are able to make changes to the code and compile and test?
That would be just a quick test to see if it changes anything. If it does any good we would have to figure out how it would affect the logging from other types of modules. Or if there should be a exception for Temperatures?
I had the capability to compile some files at one point, but I don’t believe that’s still the case. I think I had it compiling before my dog shocked the laptop and forced a reinstall
If the old forum was still up, I’d be able to find the thread where Gene posted how to set up the compiler. Not sure I’ll be successful without that info though.
It looks to me as though when a value is received via MQTT and written to the module (sensor.Temperature and sensor.Humidity), it runs the code in ModuleParameter.cs. That fires the AddValue code in ValueStatistics.cs.
To me it seems that the offending code must be (here):
// "value" is the occurring event in this very moment,
// so "Current" is holding previous value right now
if (Current.Value != value)
{
lastEvent = new StatValue(Current.Value, Current.Timestamp);
if (value == 0 && lastEvent.Value > 0)
{
lastOn = lastEvent;
lastOff = new StatValue(value, timestamp);
}
else if (value > 0 && lastEvent.Value == 0)
{
lastOff = lastEvent;
lastOn = new StatValue(value, timestamp);
}
}
If that’s the case, then I know why this code was included. I asked for it actually. Gene worked with me to update the statistics database a couple years ago. We were trying to make it work more smoothly for tracking module states (light on/off) and since X10 sends multiple instances of a command, that would really mess with the database. It messes up the lastON and lastOFF values among other things. This command will filter such instances out. However, it also filters out intentional and appropriate duplicates. I think the solution is to add a statement onto this that checks the type of module. If the data is being stored into a sensor or similar, it should write the duplicate value anyway.
In fact, the easy test for this is to modify my MQTT code to write a slightly altered value if it’s identical. Instead of 20°C 3 times in a row, I can write 20, 20.1, 20.
There is only 4 types of modules that is logged, as defined in StatisticsLogger.cs. Which one sends multiple on/off? Maybe only make exception for that one? Or could it be all 4 types?
I know the sensor, meter, and power monitor could have duplicates. I think regular modules use statistics, so the exception should be on 3 of 4 I think. I’d have to be sure about my assumption before pushing code, but I’m pretty confident that’s correct.
Then maybe more code is needed and more types, so it is possible to have duplicate values from the modules that are “real” values, and not on/off values? A module that sends on/off should maybe be of Switch type? Because that kind of module could never send a temp/hum/power value or something like that?
This is a silly question, but I’m trying to modify my code to allow a check against the old value. This is a simple work around to use as a test.
My current code is:
Action<string, string, double>
WriteToSensor = (string sensorId, string dataType, double dataVal) => {
// sensorId = module
// dataType = parameter name
// dataVal = parameter value
var module = Modules.InDomain("MQTT:Sensors").WithAddress(sensorId).Get();
if (module.Instance == null)
{
Program.AddVirtualModule("MQTT:Sensors", sensorId, "Sensor", "homegenie/generic/sensor");
module = Modules.InDomain("MQTT:Sensors").WithAddress(sensorId).Get();
}
if(module.Parameter(dataType).Value != dataVal.ToString("0.0")){
module.Parameter(dataType).Value = dataVal.ToString("0.0");
Program.RaiseEvent(module, dataType, dataVal.ToString("0.0"), "");
}
};
I need to add a statement that checks if module.Value equals dataVal and, if so, adds some small amount (0.0001). The issue is data types. module.Value is of type string and dataVal is float. I tried converting the dataVal to a string, but I can’t do a string compare correctly. I then tried to convert module.Value to a single, but that doesn’t seem to be implemented in HG. It should be something as simple as:
If you just want to test and be able to change the value, if they are the same, can’t you convert to int or ordinary double? It is only a test and you just need to add something. Doesn’t matter if it is 1 degree or 0.1.
Must be possible to convert to something useful?
I don’t think that duplicated values are filtered out.
Take a look at this line:
internal void AddValue(string fieldName, double value, DateTime timestamp)
{
if (StatisticsLogger.IsValidField(fieldName))
{
// add value for StatisticsLogger use
statValues.Add(new StatValue(value, timestamp));
}
...
// insert current value into history and so update "Current" to "value"
historyValues.Insert(0, new StatValue(value, timestamp));
}
New StatValue is added to the statValues list every time if fieldName is valid. And after that, this list is used here to calculate the average parameter value that is stored to the database:
var parameter = module.Properties[p];
if (parameter.Statistics.Values.Count > 0)
{
var values = parameter.Statistics.Values.FindAll(sv => (sv.Timestamp.Ticks <= end.Ticks && sv.Timestamp.Ticks > parameter.Statistics.LastProcessedTimestap.Ticks));
if (values.Count > 0)
{
double average = (values.Sum(d => d.Value) / values.Count);
...
