mitoXD on DeviantArthttp://creativecommons.org/licenses/by-nc-nd/3.0/https://www.deviantart.com/mitoxd/art/mito-s-Wx-Scripts-2-3-579786586mitoXD

Deviation Actions

mitoXD's avatar

mito's Wx Scripts (2-3)

By
Published:
6.6K Views

Description

Screen #2   (4) is a collection of made-up weathercasts for the demonstration of dzen’s capabilities; no online activity here, just formatting stuff.  On the left there’s the bar with the dropped down slave lines, on the right parts of the Bash code. – And supposed the relevant fonts are installed on a UTF-8 system, dzen can give a multilingual touch.  No guaranty for proper localization however. – Nonetheless there’re some limitations for becoming too fancy.  So recoloring a complete line’s background requires to type in additional whitespace.  Regarding exotic scripts the more serious one is not caused by dzen itself, but when echoing the code: While bi-directional (bidi) composing works well with graphical apps like text editors or office programs in X, non-printing Unicode characters like a right-to-left switcher get misinterpreted and become printed, and rtl expressions just “spelled backwards” left-to-right (and piped to dzen like this).

And that’s the same either the lines have been written with an X program, or the Unicode code points directly declared.  So for example here there’s no correct rendering of  ߓߡߊ߬ߞߐ߫ (Bàmakɔ̌/Bamako) via the wonderful N'Ko, or ᠬᠠᠷᠠᠬᠣᠷᠣᠮ (Qaraqorum/Kharkhorum/Karakorum) in classical Mongolian script, respectively.  On the other side it’s no problem, when rtl letters don’t change their shape whereever they appear inside a word, like Hebrew, or their different writings are hardcoded, like Arabic; see the specific Unicode blocks.  Hence (leaving further system localisation aside,) it suffices to simply c&p the expression תל אביב-יפו (Tel Aviv-Yafo) into the code, or set the explicit code points for تونس‎ (Tunis).  The latter method helps also printing Private Usage Area (PUA) glyphs or in case a font character set’s mapping differs from UTF-8 (like for the Vietnamese font).




Contents


screen #1   &   1. Specs · 2. What’s this all about …

screen #2  & .: 3.  Troubleshooting:.
  1. JavaScript
  2. Website encoding vs. system encoding
  3. The console browser does not support the character set
  4. Multilingual fonts
  5. A weather service’s reliability
  6. Completing the icon set
  7. Reducing resources consumption
screen #3   &   4. Links and References





3.  Troubleshooting


3.1  JavaScript



Wherever websites reveal the desired weather info only by some active scripts, there's hardly a chance refining the contents.  Unfortunately not only commercial sites are pretty fancy and cluttered with buttons and ads, like they’ve never heard about web accessibility.  In other words, there you can’t go to a certain place or change a scale without pressing a button with JavaScript on!  But that’d be necessary since usually text browsers can’t deal with it.  So one of the preparations is to put a script blocker on your graphical browser (see the reference list below) to see what would come out of it.


So how would you find your city’s weather on wunderground.com without JS for example?  Well, here yet you go by checking the links – so in the footer, there’s one called “Full Screen Weather”  Clicking it asks for a location, and it answers with a proper table with values, icons and more!  Works globally, and very likely in your language, too (except place names other than in Latin transcription).  And the best: the HTML source reveals both Fahrenheit and Celsius degrees, though one of them is being hidden in normal view. – On the contrary, without JS there’s no dice on the website of the Korea Meteorological Administration: Clicking any location on the Korea map defaults to Seoul then, and that’s it (any of the foreign language versions even show no reaction at all).


Yet it’s getting better at the public German Deutscher Wetterdienst.  Without JS, first there’s no clicking on any map.  However following links partially succeeds into displaying basic weather info either graphically with text, e. g. (1) “Weather + Warnings|Regional|West” (generating a new sub-site every session, not good).  Or (2) graphically embedded (new beta version).  Well, I prefer (1), which also has a fixed alias “quick link,” perfect for parsing (to be found in “News”). – Alternatively and for the record, DWD offers also an hourly free weather data service via FTP with nice (zipped) detailed text or XML table files by city or measuring station; just too sad the latest time is either the day before, or two hours late, respectively.


Some weather stations even got their own homepage. This one for example I found on the German CWOP list (see the references below).  It looks promising at first sight, but finally serves as a negative for our purpose, since there’s useful output at all without JavaScript.  Nonetheless it’s interesting, this station is somehow connected to Norwegian services. – Anyhow, though there might be a few text web browsers around that deal with JS, the next hurdle to clear is to simulate a button press, like at OWM’s for switching to °F. The best is to avoid such sites, so for the latter case, the U.S. national weather service portal a is a good start.


          [ ^ ]     [ Contents ]



3.2  Website encoding vs. system encoding



Though in the browser a website might be legible, parsing the HTML source with OS tools won’t work, if the system’s locale is not English and differs to that of the website.  Check for the first with echo $LANG, and for the latter the attribute of the charset variable in the website’s head section.  It’d be perfect, if both contain UTF-8 for Unicode support, what’s the nowadays de facto standard.  If the website does, but not your OS, then change your file system to it according to your Linux wiki.


If on the contrary the website is still written in an outdated national or regional encoding, then that’s bad luck; you might try converting the web document after downloading and before parsing though, if you think it’s worth the effort.  Actually it’s much easier to look for another website. – Alternatively, if a weather service site supports selection of other languages, look for their encoding.  The default encoding of the Korean Meteorological Administration is EUC-KR for example, so parsing garbled text is futile, while the English version is UTF-8.  It still has got other obstacles however, what leads to the next issue.


          [ ^ ]     [ Contents ]



3.3  The console browser does not support the character set


Though both file system and website match UTF-8 or another same encoding, some console browsers don’t support exotic charsets.  As a workaround, dump the website as is quietly with wget into a then created temporary folder and parse it there, like so:
# local parsing for the current temperature of Taibei 臺北, Taiwan, using national public online resources, Chinese/UTF-8 version:
trawtaibei1=$(wget --user-agent="" -q htt
p://www.cwb.gov.tw/V7/observe/real/RealData_N.htm -NO /tmp/taibei ;\
cat /tmp/taibei |grep '臺北'| awk -F\> '{print $10}'| tr -d '<
/td')

# alternative sub-site, linked to a map:
trawtaibei2=$(wget --user-agent="" -q http
://www.cwb.gov.tw/V7/observe/24real/Data/46692.htm -NO /tmp/taibei2 ;\
cat /tmp/taibei2 |grep 'td class="temp1"'|awk -F\> '{print $5}'| tr -d '<
/td'| tr '\n' ' '| awk '{print $4}')

# since w3m handles CJK very well, here’s a countercheck of the very same URL:
trawtaibei3=$(w3m ht
tp://www.cwb.gov.tw/V7/observe/24real/Data/46692.htm | awk -F\  '{print $2}'| tr '\n' ' '| awk '{print $4}')
# mind the masked double space at the first awk

Echoing $trawtaibei1, $trawtaibei2, and $trawtaibei3 will result in the same Celsius degree; anyway keep all browser scripts disabled.  And see when offline-parsing, next to locating the wanted number, the business is getting rid of HTML tags and (Windows) linefeeds, while with the browser, mainly of linefeeds.


The smart download tool wget allows to download files to a specified folder, and can force overwriting or files to be .html, if they aren’t, or be given a new name.  In case the browser’s address line misses or proper HTML file name and such, yet fetch it with wget and just the given URL.  For further details I recommend reading its manual. – It’s a matter of taste and convenience, how you track the weather data.  The curl method however might help reducing web traffic footprint.


          [ ^ ]     [ Contents ]



3.4  Multilingual fonts


As far as I experienced this, dzen can take only one font at a time (in comparison to e. g. rxvt-unicode).  So depending on what different writing systems should be displayed, better have them combined within the same font.  Usually “foreign” fonts at least contain a standard set of ASCII characters, i. e. the Latin alphabet including punctuation and some additional symbols.  And it’s even worse with symbol fonts, where symbols are at the character range positions of the standard alphabet, such as Webdings and Wingdings.


In the course of internationalization (TrueType and OpenType) Unicode fonts have been developed to a greater extend, this article gives an introduction.  For the sake of readability, I recommend sans serif a.k.a. Gothic fonts.  As always, fonts can be installed as packages as well as manually according to your Linux distro’s wiki. – In case you still wouldn’t find a single font, there’s still a way with using several dzen bars and the -fn option.  Nonetheless occasionally there might appear small but ugly gaps between the bars, when flushing the same direction.


Even in case with ready Unicode fonts and UTF-8 support, there’s still a risk for foreign writing systems (like right-to-left) not displaying properly.  While such phrases appear correctly in X programs, when echoed on X terminals, any letter show up as isolates regardless of its position in a word.  If anybody knows a workaround or a solution for that, please drop me a line.


          [ ^ ]     [ Contents ]



3.5  A weather service’s reliability


The minute you wonder, why the weather right outside your window is different to that of your place displayed on your screen, will make you take some more general considerations about a service’s or station’s choice.


Sure the location of a weather station is of main importance.  So it’s in its nature of things that data given by a private measuring station at a central main road often differs from that of an airport on the outskirts.  ICAO is the most common standard for addressing international airports.  While the latter carries an ICAO ID, it’s always a good idea to double-check a station’s geographic coordinates.  Meteorological sites can do this, or at least link to online maps; so do Wikipedia articles about airports.


Be also sure of a (foreign) location name’s correct spelling.  Again it’s all about conventions, note the several versions of e. g. “Pusan.”  Supposing it’s the first hit at yr.no for South Korea’s metropole 부산, that’s nowadays officially transcribed as “Busan.”  The details of the location link reveal even more.  You might then want to check the Geonames-ID.  Alternatively look for it at another service, like the formerly mentioned “KNYC,” which directly leads to the NY Central Park’s measuring station (and a population of 0, lol~).


Further mind there’s a wide range of refresh rates. It does not make sense to fetch new data more frequently than a station or a service provides them.  If an update interval is not apparent, it has simply to be tested. – yr.no offers a wide range of useful stuff for free which could be easily parsed and processed, including a huge downloadable icon set and XML forecast files.  On the other side, it’s extremely rarely updating the data only a few times a day.  OMW’s on the other hand is doing this every few minutes, and so do many CWOP clients.  It eventually appears, private services are getting more often updated than public.


For Jane & John Doe user, it’s always a good idea to ask who is feeding a website with what information for whom, and why.  I’d say it’s nearly impossible to say though, if governmental and scientific weather services are more to be trusted compared to private.  Or, (own) national to regional or international ones.  Or, if weather data are maybe spoofed, or whether your IP gets recorded. – What is possible to approve however is the chance of high technical quality.  At weather service sites there will be references to scientific and other partner organizations.  Also there’re station lists, while certified stations vice versa will proudly mention their gear as well as any network memberships, since usually they have to meet certain requirements, as for CWOP; in comparison anyone and any station owner can join the network of OWM even for free, while reliability also vaguely depends on the pricing.


In the end, even if you use a reputable network to retrieve weather data from an ICAO registered station, there’s no warranty when an exotic location lacks of working infrastructure, is closed at night, or whatever.  Once I experienced a Notrh Korean international airport was permanently at 37° C with a clear sky, while an alternative Chinese one within the region was occasionally down.


          [ ^ ]     [ Contents ]



3.6  Completing the icon set


Here one has to face two problems: There’s no weather service site except OWM, that grants access to all icons at once.  So that’s also why I actually prefer OMW’s list of weather conditions as some sort of template, though it contains a small icon collection compared to many others, and icons are assigned multiple times. – Assuming you want to mirror another weather service’s icon set one-to-one, there's no other way than to download icons just as they appear on the website, if any (e. g. NOAA has none).  Unfortunately even if a server’s icon folder URL can be figured out, direct access is often forbidden. – To speed things up, look up the weather at other global locations, if the weather service supports this.


The other is not less severe: There’s neither a complete list of condition terminology for the public, as well as the services’ terminologies vary and differ in quantity.  So they are hardly exchangeable, and hence the creation of a complete beforehand term–icon correlation is nearly impossible, in a sense of one list for any service.  That’s a point for staying with the weather service’s icon set.


Anyway there’s a quite sophisticated workaround though.  It’s got to do with standardized surface weather observation.  And that’s a professional business, coming with diverse technical terms of its own: For such purposes the already mentioned ICAO uses the METAR code, a complex array of abstract looking letters and numbers combinations.  This means no icons here, but all necessary parameters can be easily skimmed.  See this example of a recent observation history for New York City, Central Park; obviously there’s a tight connection between weather and sky condition.  As a consequence, you’re encouraged to compile your own set of icons based on METAR then.


          [ ^ ]     [ Contents ]



3.7  Reducing resources consumption

Using curl [link] instead of wget-ing or browsing websites just for dumping values helps keeping online traffic and local memory usage low:

weatherservice1="https://web.kma.go.kr/eng/index.jsp"     # Seoul, Korea
tempraw=$(curl -s ${weatherservice1} | grep -m 1 second | awk -F\℃ '{print $1}'| awk -F\> '{print $5}'|tr -d '\n')
         # filtered local air temperature; needs curl installed
Depending on a weather service’s website structure the other methods might still look more convenient though (example in Bash):
$ # Berlin, Germany (Deutscher Wetterdienst)
$ berlinweather="https://www.dwd.de/DE/wetter/wetterundklima_vorort/berlin-brandenburg/berlin_tempelhof/_node.html"
$ berlintemp=$(w3m ${berlinweather}|grep -m 1 Temperatur | awk '{print $2}' |tr -d '\n')
$ echo $berlintemp
4
To get the same result with curl one needs to get a bit deeper into grips with it.


An icon can be retrieved like so, downloading it only if it doesn’t already exist:

weatherservice2="https://www.yr.no/place/South_Korea/Seoul/Seoul/"  # Seoul sky condition service
sky=$(curl -s ${weatherservice2} | grep "og:image"| awk -F\/ '{print $7}'| awk -F\. '{print $1}' |tr -d \"\\n)
skypath=https://www.yr.no./grafikk/sym/v2017/png/200/     # with courtesy of yr.no weather forecast service
skyPPiconpath=${HOME}/my/skyicons/path/
skyPPicon=${skyPPiconpath}${sky}.xpm                      # XPM for piping to dzen2

if [ ! -f "${skyPPicon}" ] ; then
     convert ${skypath}${sky}.png -resize 30x $skyPPicon  # needs imagemagick installed
fi
This way assumes the relevant icons are not hidden by JavaScript.



          [ ^ ]     [ Contents ]






.: mws Screen #1 :: UI Design :: Code Corner :: mws Screen #3 :.



Comments welcome!

Image size
1024x768px 515.69 KB
Comments6
Join the community to add your comment. Already a deviant? Log In
default-cube's avatar
Just a note, you don’t need to save the wget output to a temporary file, you can use “-O -” to write it direct to standard output. This saves the additional cat step.