Inaccuracy with OpenWeatherOrg data

soxfan1966 · April 21, 2020, 4:56pm

Hello X10 world, hope all is well. I’ve noticed some inaccuracies with the forecasts for my Weather Widget from OpenWeatherSource lately and finally had a chance to dig into it.

I’ve been delving into the world of API calls at work so I thought it was a good chance to play around with some of the API calls that HomeGenie was using for the weather.

Essentially, the calls to OopenWeatherMap are done as follows:

Current Conditions:
“https://api.openweathermap.org/data/2.5/weather?q=” + location + “&lang=” + language + “&units=metric&appid=” + apikey

so:
https://api.openweathermap.org/data/2.5/weather?q=Charlton, US&lang=EN&units=metric&appid=

That returns:
{“coord”:{“lon”:-71.97,“lat”:42.14},“weather”:[{“id”:804,“main”:“Clouds”,“description”:“overcast clouds”,“icon”:“04d”}],“base”:“stations”,“main”:{“temp”:8.9,“feels_like”:0.48,“temp_min”:8,“temp_max”:11.11,“pressure”:1001,“humidity”:65},“visibility”:16093,“wind”:{“speed”:9.8,“deg”:180,“gust”:16.5},“clouds”:{“all”:90},“dt”:1587484856,“sys”:{“type”:1,“id”:5231,“country”:“US”,“sunrise”:1587463015,“sunset”:1587512138},“timezone”:-14400,“id”:4932823,“name”:“Charlton”,“cod”:200}

Then another call is made for the forecast:
“https://api.openweathermap.org/data/2.5/forecast?q=” + location + “&lang=”+language+"&units=metric&appid=" + apikey

so:
https://api.openweathermap.org/data/2.5/forecast?q=Charlton, US&lang=EN&units=metric&appid=

that returns a bunch of stuff - and if you look at the data and specifically this field:
“dt_txt”:“2020-04-22 06:00:00”

you can see that it appears to be a forecast in increments of 3 hours

In the Program code for OpenWeatherMap, there is this:

var nextDay = DateTime.UtcNow.AddDays(1); nextDay = nextDay.Date + new TimeSpan(12, 0, 0);

I believe that is setting the time portion for the date/time stamp to be noon? Or is it 12 hours from the current time ?

The code then appears to try and find the matching date/time entry for the next 3 days, matching the date and time. I’m not 100% certain but I am wondering if the way this is trying to match may be off or not. I would assume the forecast would just be at a set time every day for the next 3 days ?

soxfan1966 · April 23, 2020, 12:13pm

I played around with the Program code a little yesterday, comparing the results against the hourly data that Wunderground weather uses (here is an https://www.wunderground.com/hourly/us/ma/charlton/KMACHARL11 for my town for hourly for a day as well as this one which shows a graph with hourly forecast for the next few days
https://www.wunderground.com/forecast/us/ma/charlton/KMACHARL11

By changing this line of code in the Weather program:

var nextDay = DateTime.UtcNow.AddDays(1); nextDay = nextDay.Date + new TimeSpan(12, 0, 0);

and changing 12 to 15, I get the data from 3pm. Changing to 18 gives me the data from 6pm. Using 6pm, that seems to more closely match the forecast high temp for the days from weather.com.

I think the most accurate solution would actually be to take all of the data returned from the OpenWeatherMap API for the forecast and then comb the hourly entries for the day and pull out the min and max temps from the whole day and then use those values. However, to get the overall weather forecast (i.e. rain. snow, partly cloudy, etc) you still need a single entry so not sure if that is worth it or not.

Overall, at least for me, I don’t think noon makes sense to use as the time of day to get that day’s forecast from - its not the coldest nor warmest part of the day so what you end up with is what looks like inaccurate forecasts for the upcoming 3 days. I think 3pm probably makes the most sense since that is closer to when the warmest part of the day is (though that may also depend on the time of year).

At least knowing that the forecast info is either from noon (using the current Program code) or some other time if you change it allows you to put it in perspective.

Interested in what others may think. Thanks

bkenobi · April 25, 2020, 4:37pm

The max temperature is typically between 2-4pm. The min would be around dawn (before solar heating kicks in). This is obviously assuming that there is not an afternoon storm, temperature inversions, other weather phenomenon that affect a portion of the day. So if you want to find the min/max temps, you would probably want to poll at 3pm for the max temp and whatever hour marker is after dawn for the coolest time which depends on the time of the year. And for that matter, 3pm may vary seasonally as well (not sure). At my latitude (Seattle), we have daylight between ~8am-3pm in December and 5am-10pm in June.

So maybe the best solution would be to simply take the time of sunrise and sunset to pick your polling times. Select the full hour after sunrise/dawn as your cold time check. For the hottest time, probably average sunrise/sunset and round to the next hour. For me, that would say that in summer I would poll at 5am and (5+22)/2=4pm and in winter 8am and (8+15)/2=12noon.

Assuming you have the solar altitude program running, you already have those values. If not, I assume the weather data you are grabbing pulls that also so you could just use their values.

soxfan1966 · May 9, 2020, 7:07pm

Good afternoon all,

Finally had some time to play around with this (what else to do when confined to home and its snowing in May).

Anyway, what I did was update the OpenWeatherMap program and widget so that they compile two sets of data for each forecast day. The first, the morning, is from 9am. The second, the afternoon, is from 3pm. Theoretically, those could be updated as desired and/or additional segments added.

Here is what I have for the widget as of this afternoon, still need to do some debugging.

I also spent some time trying to find a way to fix the temp_min reporting issue (OpenWeatherMap returns the same value for the min temp as the max temp, something tuicemen had pointed out and modified the widget to include the humidity instead).

soxfan1966 · May 10, 2020, 11:57am

Good morning,

I made some more tweaks last night and this morning. I added another row of data - for evening. So now I have morning, afternoon and evening. Originally I had morning at 9am and afternoon at 3pm but this morning I decided to change as follows:
morning = 6am (I am usually up by then so its more useful to me than 9am)
afternoon = noon (I may use 3pm still - will see)
evening = using 6pm.

I was thinking I could add options for these values and allow a choice for the morning as either 6am or 9am, afternoon as either noon or 3pm and evening as either 6pm or 9pm.

I may do that today.

Also - a bug (or maybe an oversight) in the original version of the widget and carried over to mine. The last update info at the bottom is quirky - the date can actually end up being tomorrow in the late afternoon. What I think was happening is that once the next forecast time is past today it will use tomorrow’s date (but still the current time) - its all in the way the string for last update is built. So in mine I removed the date portion. The days being forecast will also get thrown off - so late in the afternoon when the last update date turned to Sunday even though it was still saturday, it was doing the forecast for Monday, Tuesday and Wednesday and not Sunday, Monday and Tuesday. I have fixed this as well - though now it will show the forecast for Sunday even when it is Sunday, at least until the last forecast time if met.

Here is my latest version of the widget:

Tuicemen · May 10, 2020, 12:29pm

Nice, perhaps you can add your findings to the issue I posted on the Open weather map https://github.com/genielabs/HomeGenie/issues/382
It may motivate Gene to fix some of it the next time he creates a new build.
As a suggestion since you’ve included the condition icons for each time you may wish to remove the conditons wording from under the date as it only applies to the morning.

soxfan1966 · May 10, 2020, 12:37pm

I will do that once I have it finished up - Thanks

soxfan1966 · May 10, 2020, 12:52pm

As suggested I removed the forecast test - will post on the HomeGenie site once I have it working with the options etc and making sure it works as expected.

Tuicemen · May 10, 2020, 2:50pm

Nice, 1 more suggestion regardless of whether or not you add the optuon for the end user to change the times. A lable of morning, afternoon, and evening may help clarify things.

soxfan1966 · May 10, 2020, 3:32pm

Was thinking I would display the time for the forecast (i.e 6am) or such so want to get the option fields in there.

I also use Fahrenheit as opposed to Celsius so some of the code is using that (one of the bugs you reported with the standard). And I also employ rounding. Once I’m a little more finished I can share the code if you are interested.

Tuicemen · May 10, 2020, 3:34pm

By all means do share I’m sure others would be interested as well.

soxfan1966 · May 11, 2020, 12:34am

Here are the latest changes from this evening - a few fore tweaks to code (the morn, afternoon and evening times are based on user defined choices as is the use of the F vs C and I hope to add in the option to round the temps or not before sharing).

Tuicemen · May 11, 2020, 11:09am

Looking good the only thing else you may wish to add is UV. I think that is a separate pull from OpenWeather but still free. If I remember Gene had commented in the code for UV but it may have been just a to do comment though🙄

soxfan1966 · May 13, 2020, 2:42pm

Been working on the widget and associated changes a bit more the last few days.

One problem is that the hourly weather from OpenWeatherOrg comes in at midnight, 3am, 6am, 9am, noon, 3pm, 6pm, and 9pm - every 2 hours.

There is an existing bug in the standard OpenWeather widget where after noon (which is the default time that it pulls in the forecast info for), it skips a day ahead. For example, later this afternoon you would see it report May 15th as the first day instead of May 14th as the next day, and then the last updated day / time at the bottom also shows the wrong day.

What I tried to do is have the first forecast day be the current day - makes sense especially with having a morning, afternoon and evening forecast. But it starts cutting off the forecast times that have passed - so right now, the first forecast time for today is at 3pm (afternoon). The way the match logic works its throwing off the days and times for each of the 9 instances I am trying to report (days 1-3 are morning, 4-6 are afternoon and 7-9 are evening, where days 1, 4 and 76 are all the first column (today or tomorrow depending on the setup).

So here is what I am thinking - I am going to try and use the start still at today - but if a match for that time slot is not found then the widget will just be blank in that area. So you may see the morning for today if its available then it would not be, same for the afternoon, etc. That’s going to take some reworking of the logic.

I also have added a way to use the F or C icons via an option and also the rounding of the temperatures to whole numbers or not.

I was attempting to get that to also fix the issue with the temp HG shows in the upper right corner but its a tad wonky. By default, HG pulls in its weather data as metric and then if you have imperial (F) selected it converts it. I have tried to change that to pull in the imperial if you have F selected but doing that throws off the main HG temp display (yesterday it was 213 degrees at my house ).

This is the inner HG code where the main temp is calculated / displayed:
$$.GetLocaleTemperature = function (temp) {
var temperatureUnit = HG.WebApp.Store.get(‘UI.TemperatureUnit’);
if (temperatureUnit != ‘C’ && (temperatureUnit == ‘F’ || HG.WebApp.Locales.GetDateEndianType() == ‘M’)) {
// display as Fahrenheit
temp = Math.round((temp * 1.8 + 32) * 10) / 10;
} else {
// display as Celsius
temp = Math.round(temp * 10) / 10;
}
return (temp * 1).toFixed(0);
};

I cannot find where this function is being called from (so where is the temp being passed in). I was hoping to be able to update the Sensor.Temperature value within my updated widget so that if I pulled it in as a F (i.e. imperial) it would convert it to C so that HG would be able to correctly display it without any modifications to that code (I have a pull request about rounding, etc and doing it within the widget would make it so that change was not necessary).

But I can’t seem to figure that out. My current solution is in my updated widget program to pull in the weather data as metric and store it in the temperature field as the standard widget does, but add a second set of weather data based on the user preference and then use those values to display in the widget only. With that approach, it works but if I take a HG update then I need to go back and edit the system file where that code is to restore my rounding change.

So that’s where I am at with things as of this monring.

Petediscrete · May 13, 2020, 5:50pm

Unfortunately many other home automation systems suffer the same fate too. You will see this happen when you modify a plugin with your own code. As soon as you update the system you are left with the default settings. This can be a pita with regular update cycles.

On the plus side I doubt you will have to worry too much about this in HG. I’m pretty sure you won’t see any more updates to the core system so you can happily modify the code any way you want safe in the knowledge you won’t have to go through the same procedure every time.

Modifying OpenWeather has given you a good insight to how HG is coded. You’re stepping outside the basic user zone so you can see how flexible the program is. Using the Program API really extends its usability.

Weather forecasting really isn’t my thing but you’ve managed to tailor OW to suit your own purposes. You’ll soon be able to replicate your efforts with the other modules in HG.

Hopefully some of the more seasoned users like @bkenobi will be able to give you feedback on your efforts.

Nice work!!!

Tuicemen · May 14, 2020, 11:19am

For your time forecast you might try an if statement.
If current time is greater then the defined display time don’t change the value.

soxfan1966 · May 22, 2020, 11:02am

Good morning all,

I have been messing around with the standard Weather Widget and associated program for the last several weeks. I had noticed an issue where the forecast for the left most column would be off later in the afternoon (it was advancing to the next day). There is a post elsewhere that gets into some of the details, but I wanted to share what I have now with people in case they wanted to try this out and provide any feedback.

My updated version provides 3 separate forecasts for each day - morning, afternoon and evening. This data still comes from OpenWeatherMap. The standard widget uses the 12pm (noon) time to get the forecast data. My version allows you to pick between 2 times for each of the daily forecasts:
morning = 6am or 9am
afternoon = noon or 3pm
evening = 6pm or 9pm

Note: The hourly data provided by OpenWeatherMap is every 3 hours, starting at midnight (so midnight, 3am, 6am, 9am, etc).

My version also starts with today, at least while the forecast data for today is still available whereas the standard widget starts with tomorrow, and once past noon it would advance to the following day (the Last Update date on the bottom of the widget would actually display tomorrow’s date when this happens, even though its still today).

When the forecast data for today (morning or afternoon) is no longer available, that area in the widget will appear blank. The attached image shows this for the morning forecast for today. Once the evening forecast data is not available, then it is advanced to the following day (and the other 2 days also move forward 1 day).

Note: if you look at the code, there are a number of debugging widget DIVs and Program.Notify calls that I commented out but left in the code, if you uncomment those you can see the date/time each forecast for each day was last updated, as well as an option for Debugging mode which allows you to see what date/time for the forecast data is matched with the displayed day. There are 9 days of data, 1-3 are the morning, 4-6 are the afternoon and 7-9 are evening. Going to Configure / Widgets and pulling up the new Weather Widget (homegenie/generic/weather) and then click the hamburger icon next to the “bind to module” will allow you to see the Debug fields as long as the Debug option is enabled.

Other Changes:

I use the same humidity icon on the widget as HG does on the Dashboard, it looks like 2 raindrops.
I am displaying the feels like low forecast temp (there is a known issue with the OpenWeatherMaps data where the high and low temps are the same - no idea why, but there is no getting around it). The icon for the forecast low is the down arrow followed by ~ to attempt to identify its an approximate low.
the current forecast string has been updated to displayed as mixed case as opposed to lower case: So “Clear Sky” and not “clear sky”, no particular reason other than it bugged me.
There is a new Widget option for “Fahrenheit Enable” - this will correctly (I believe) allow the display of C or F data and icons without the need to update the standard widget code (a reported bug).
There is a new Widget option for “Temperature Rounding Enable” - if set, it will round and truncate the temp values to whole numbers.

the display of the temperature in the HG Dashboard while using the widget data is done in its own code, I attempted to correct the issue of rounding / truncating with this same change but not convinced it worked, there is a post on the HG board with info on how to make the change to the system programs to round that temp if needed. Also, the conversion of the temp for F vs C is also in the HG programs, so I had to essentially leave the retrieval of the temperature field used by HG in place and use a new set of fields for the widget, where the widget specific ones are based on the imperial or metric based on the choice of the Fahrenheit enable whereas the HG used ones are always metric and then converted.

Precipitation Rate option was intended to allow for user choice if the precipitation info was based on the 1h or 3h values - unfortunately we have not had recent rain for me to verify this but it should work.
I mentioned the forecast time options above already (morning, afternoon and evening time choices)
The Precipitation Last 3 Days option currently does not do anything (I was / am working on a way to try and display the precipitation for the last 3 days, thought it would be useful this time of year as I need to monitor how I water my lawn).

I think that’s it - the code for the program and the widget, while still mostly based on the original, has been cleaned up with my changes and I tried to comment where it made sense to help explain things.

Last note: If you upload this new program it will be program number 1030 and I added “Soxfan1966 version” to the name. Its possible for you to have both the standard and my version running (I still have the original version installed but not running). If you chose to inactivate the standard one, and you use tuicemen’s Weather Alerter program then you will need to update that so that it references the new version of the weather program.

I am also going to post this over on the Home Genie Club Forum. Feedback and suggestions welcome.

Thanks

1030-OpenWeatherMap_-_Soxfan1966_Version.hgx (21.3 KB)
homegenie_generic_weather.zip (5.0 KB)