AVS Forum banner

601 - 620 of 631 Posts

·
Registered
Joined
·
111 Posts
These were the kinds of things I expected from Pigs on Windows.
Glad you got it sorted quickly and posted results.

Meanwhile, on the Pi front:

"PiGS Start Time": "2020-09-22 11:08:19(EDT)",
" PiGS Up-Time": "20 days, 5:41:48.749305",
" PiGS_Version": "V1.00 2020-Aug-14",
" Builder Alive": true,
" PiGS Directory": "/home/pi",
"Service Build Info": {
"Services_Built": "2020-10-11 02:24:20(EDT)",
"Total Channels": 21
 

·
Registered
Joined
·
1,712 Posts
Tim that was the problem! Hope others learn from my bad experience!

What is strange is it picked on the pigs.exe file for v1.12 but had no issue with the pigs.exe file for v1.11? A bit of a mystery!
You guys must be using something other than McAfee AV Plus, because that is just happy as a lark with the pigs.exe in both versions. That's not to say that all AV is happy - Google's Virustotal gets 9 hits out of 90 AVs, and it's well documented that binaries generated by pyinstaller have this false-positive problem.
 

·
Registered
Joined
·
1,712 Posts
i just renamed my pigs.exe to p.exe and ran the batch file pigs.cmd and thats what you get.... a repeating chain of pigs 80's
i remember now, i found my pigs.exe in the virus scanner quarantine
you need to add an exception to your virus scanner for pigs.exe so it does not delete it.
the python 'compiler' that makes the windows stand-alone package is known to produce false positives with many scanners....
To make it more obvious if this ever happens, and if we ever pop out a new version, I think I'll change the PiGS.cmd file to say "pigs.exe 80" instead of "pigs 80". That way it won't run PiGS.cmd recursively if pigs.exe is missing (bummer that filenames in Windows are case insensitive; the Unix/Linux folks got it right), and print a meaningful error message instead.

ADDED: And for anyone currently using 1.11 or 1.12, just edit PiGS.cmd to say pigs.exe 80 and instead of looping like that if pigs.exe goes missing, it should just say pigs.exe not found.
 

·
Registered
Joined
·
7,550 Posts
bummer that filenames in Windows are case insensitive; the Unix/Linux folks got it right
I don't think I agree - case-sensitive file names are much easier to mistype, resulting in hard-to-diagnose errors. They probably made sense when Unix was first being developed, but computers were much less powerful then: it was over 40 years ago!

This issue actually stems from a Windows quirk inherited from the days of MS-DOS: you can have two "executable" files with the same name (pigs.cmd and pigs.exe, and you could even have pigs.bat and pigs.com too, if you really wanted to push it). The DOS and Windows command interpreters try the various valid executable extensions until a matching file is found. Linux shells don't do anything like this, relying instead on the 'x' permission flag, so they're immune from this sort of issue.
 

·
Registered
Joined
·
2,188 Posts
To make it more obvious if this ever happens, and if we ever pop out a new version, I think I'll change the PiGS.cmd file to say "pigs.exe 80" instead of "pigs 80". That way it won't run PiGS.cmd recursively if pigs.exe is missing (bummer that filenames in Windows are case insensitive; the Unix/Linux folks got it right), and print a meaningful error message instead.

ADDED: And for anyone currently using 1.11 or 1.12, just edit PiGS.cmd to say pigs.exe 80 and instead of looping like that if pigs.exe goes missing, it should just say pigs.exe not found.
Thanks Frank70! I made the change and it works fine!
 

·
Registered
Joined
·
1,712 Posts
I don't think I agree - case-sensitive file names are much easier to mistype, resulting in hard-to-diagnose errors. They probably made sense when Unix was first being developed, but computers were much less powerful then: it was over 40 years ago!
Well, we'll have to agree to disagree. But here is an interesting historical tidbit in this vein that perhaps you didn't know:

CP/M, a predecessor of DOS for 8086, 8080 and Z80 processors, has case-insensitive file names, or so it appears. But that's not really so. CP/M-80 2.2 is quite capable of writing lower case characters to a disk directory, but all the CP/M commands and utilities that create or manipulate a filename will always translate any lower case characters to upper case before writing them to the disk directory. But the command MBASIC (the program MBASIC.COM, microsoft BASIC for CP/M) can save one program as pigs.BAS, an entirely different program as PiGS.BAS, and a third different program as pigs.bas; and subsequently load any one of them. I assume it can do the same for data files. The lower case characters in these names will stay lower case in the diretory and will display that way when a CP/M dir command is executed. Since all other CP/M commands translate file names to upper case, no other CP/M command can access a file with lower case letters, not even era (erase). To delete such files, it is necessary to go back into MBASIC and use its kill command; e.g.: kill "pigs.bas".

I suspect this unusual behavior of MBASIC may be because it accesses the disk at a layer below the normal CP/M BIOS calls. Or maybe it was just to drive the users a little crazy.

Ok, now back to our regularly scheduled PiGS programming ...
 

·
Premium Member
Joined
·
257 Posts
I don't think I agree - case-sensitive file names are much easier to mistype, resulting in hard-to-diagnose errors. They probably made sense when Unix was first being developed, but computers were much less powerful then: it was over 40 years ago!
I've never had such a problem in 30+ years of Unix/Linux/C/C++ programming. And it has nothing to do with how powerful computers were 40 years ago. Upper-case is upper-case. Lower-case is lower-case. If, as a programmer, you can't differentiate, you shouldn't be a programmer. I'd HATE to see the code produced by a programmer who couldn't tell the difference.
 

·
Registered
Joined
·
7,550 Posts
I've never had such a problem in 30+ years of Unix/Linux/C/C++ programming. And it has nothing to do with how powerful computers were 40 years ago. Upper-case is upper-case. Lower-case is lower-case. If, as a programmer, you can't differentiate, you shouldn't be a programmer. I'd HATE to see the code produced by a programmer who couldn't tell the difference.
You're certainly entitled to disagree, but insulting straw-man arguments aren't exactly appropriate.
 

·
Registered
Joined
·
21 Posts
If, as a programmer, you can't differentiate, you shouldn't be a programmer.
But everyone can differentiate. That's not the point. Making assumptions about default OS behaviour was what caused the issue.
 

·
Registered
Joined
·
1,712 Posts
But everyone can differentiate. That's not the point. Making assumptions about default OS behaviour was what caused the issue.
Well, not quite... the assumption was that .exe took precedence over .cmd, which is indeed the default OS behavior. What was not so expected was that the .exe would just plain disappear on its own.

That's the fault of the virus scanner, and as I've said before at least 9 and possibly more virus scanners throw a false positive on these compiled python 3 programs. Submitting the .exe to each of the offending virus scanning companies so they could fix that would be quite the undertaking. Or, I suppose it's possible that python.org is unwittingly party to circulating a real virus, though curiously all the 9 Virustotal virus scanners identify the .exe as harboring a different virus:

SecureAge APEX Malicious
Cyren W32/S-08ce9f36!Eldorado
eGambit Unsafe.AI_Score_97%
FireEye Generic.mg.6b906fd5cfe183ca
Ikarus Trojan.Python.Psw
McAfee-GW-Edition BehavesLike.Win32.Generic.vc
SentinelOne (Static ML) DFI - Suspicious PE
Sophos ML Generic ML PUA (PUA)
Symantec ML.Attribute.HighConfidence
It's also possible that updating to python 3.9.0 from python 3.8.5 might result in a fix. Python 3.9.0 was released very recently.
 

·
Premium Member
Joined
·
234 Posts
Discussion Starter #611 (Edited)
It's also possible that updating to python 3.9.0 from python 3.8.5 might result in a fix. Python 3.9.0 was released very recently.
or even a rollback to 3.7 (which is what I develop on), .6 or .5
but we know its an issue and you just have to exclude the file from the virus scan

or maybe submit it to one place... python.org. whatever is causing that signature could probably be changed.
 

·
Registered
Joined
·
28 Posts
There was a discussion about 2 years ago on the main DVR+ Owners thread about recorded shows having two different descriptions (DVR+ bug) and it would be interesting to know if it happens on PiGS.

Basically, go to your recorded programs list and hit [Info] on a program. You will get the proper Description received from your guide provider. View [Options] and you'll see the same description. So far, so good.

Now start playing the recording, then hit [Info] and you will sometimes see a completely different description than your guide provider gave you. The description is still correct for the recording (i.e. it's not from some other show) but the text will be completely different.

Bonus points to anyone who can guess where the second description comes from? Since it's not coming from the guide server, could it be from PSIP?

Example 1: neXt S01E01 (aired 6-Oct-2020 @ 9:00pm on Fox)
Guide server description: Silicon Valley pioneer Paul LeBlanc joins forces with Special Agent Shea Salazar ...
2nd description not from guide server: A series of disconcerting technical mishaps suggest a worldwide ...

Example 2: Departure S01E02 (aired 15-Oct-2020 @ 10:00pm on Global Montreal)
Guide server description: The team pursues leads as they wait to see if the surviving witness will regain ...
2nd description not from guide server: The team wait to speak to an unconscious witness. The aircraft is located.
 

·
Registered
Joined
·
28 Posts
Have you use DVR+Lister pachinko's program ? Does it show both ? Check dedicated thread to the program...
I don't use DVR+ Lister since I run linux exclusively, but I did poke around with a hex editor.

The record_event_index file only shows the information from the guide server (as expected). The second mysterious description does not appear anywhere in the file. The record_event_index file is the main file parsed by DVR+ Lister.

I did find something interesting though. Each recorded show (.ts file) also has a corresponding .log file stored on the hard drive. The .log file for the two examples above contain three descriptions. The first field is the description from the guide server. The second description contains the mystery text, and the third description seems to be the description for the next show on the grid.

In the case of NeXt that was in my original question:

NeXt S01E01 (aired 6-Oct-2020 @ 9:00pm on Fox)
1st field in .log file - Guide server description: Silicon Valley pioneer Paul LeBlanc joins forces with Special Agent Shea Salazar ...
2nd field in .log file - 2nd description not from guide server: A series of disconcerting technical mishaps suggest a worldwide ...
3nd field in .log file - description for next show in grid: Fox-44 News airing @ 10:00pm

I think I'm going to post in the main DVR Owner's thread as these technical details are not PiGS specific. The only reason I posted here in the first place is I wanted to know if PiGS showed the same behaviour.
 

·
Registered
Joined
·
1,629 Posts
I don't use DVR+ Lister since I run linux exclusively, but I did poke around with a hex editor.

The record_event_index file only shows the information from the guide server (as expected). The second mysterious description does not appear anywhere in the file. The record_event_index file is the main file parsed by DVR+ Lister.

I did find something interesting though. Each recorded show (.ts file) also has a corresponding .log file stored on the hard drive. The .log file for the two examples above contain three descriptions. The first field is the description from the guide server. The second description contains the mystery text, and the third description seems to be the description for the next show on the grid.
FYI, member SirCrow recently has DVR+ Lister running Ubuntu 20.04 under Wine/PlayOnLinux (see DVR+ Lister for Channel Master DVR+). Everyone interested in running DVR+ Lister on a Linux computer should investigate his post. I've already had a "conversation" with SirCrow to the extent of my knowledge of running in Linux (which is VERY little), so someone else may be able to answer a problem he is having with the DVR+ Lister Converter.

The DVR+ gets the mystery description from the log files, and that comes from PSIP, which comes OTA for each channel received (not related to Zip Code). The guide description is stored the REI file.
 

·
Registered
Joined
·
1,712 Posts
Here's something a little unusual I noticed today running PiGS on my Pi Zero W.

My DVR+ #1 (192.168.1.194) started requesting its daily listing refresh at 11:14:49 EDT and made its 88th request at 11:24:28 EDT. The 1st request resulted in 112 listings being sent to the DVR+ for 10/17 12:00:00Z. The 88th request resulted in 118 listings being sent to the DVR+ for 11/01 08:00:00Z, but it never made requests number 89 and 90.

About an hour later, my DVR+ #2 (192.168.1.189) started requesting its daily listing refresh at 12:09:34 EDT and made its 90th request at 12:15:55 EDT. The 1st request resulted in 96 listings being sent to the DVR+ for 10/17 16:00:00Z. The 90th request resulted in 116 listing being sent to the DVR+ for 11/1 12:00:00Z.

I'm curious why DVR+ #1 only made 88 requests and not 90. Somehow, the times just don't seem right, but a scan through the guide did not seem to show any holes. EDIT: I notice that between requests 38 and 39 there is an 8 hour gap instead of a 4 hour gap; and between requests 72 and 73 there is an 8 hour gap also. Yet there don't seem to be any listings missing in the guide during the corresponding time slots.

Yesterday, both DVR+ units made 90 requests, but each slightly earlier. The log covers yesterday and today.

The log and status files are attached.
 

Attachments

·
Premium Member
Joined
·
234 Posts
Discussion Starter #618 (Edited)
I'm curious why DVR+ #1 only made 88 requests and not 90. Somehow, the times just don't seem right, but a scan through the guide did not seem to show any holes. EDIT: I notice that between requests 38 and 39 there is an 8 hour gap instead of a 4 hour gap; and between requests 72 and 73 there is an 8 hour gap also. Yet there don't seem to be any listings missing in the guide during the corresponding time slots.
Yesterday, both DVR+ units made 90 requests, but each slightly earlier. The log covers yesterday and today.
Here is what I notice.... your build today did not complete
"Schedule Build Info": {
"DaysLimit": 17,
"BuildStartTime": "2020-10-17 10:04:28(EDT)",
"Build_End_Time": "Rebuilding...",

"Build_Progress": "43%",
"Programs": 0,

There was a build exception after 12.3, so maybe on your chan 17.1:
LS:BLS: Added 803 programs for channel 12.3, RfChan=13 (StationID=51000)​
THRD:BUILDER: Build Schedule Exception occurred!
THRD:SetWDogBuildTimer: Auto Build time set for 2020-10-18 10:05:44(EDT).​
THRD:BUILDER: Queue is Empty, releasing SDResourceLock​

So it would appear there would be BIG holes on all chans > 12.3 for TODAY

So you could do a 'SaveSettings' build and see if the exception happens again, if so, you could set the log level to debug and 'savesettings' again to see if something else gets captured. and you have your stdio captured too, right? anything in there?

with the background build thread the build is now wrapped in the try/except, so the thread appears to not have died so it should recover, but why did it happen...
maybe some bad data (empty?) data from SD i'm not handling correctly???

As far as yesterday and the skipped blocks... the ip/port did not skip a beat so it doesnt appear to be a dropped packet. that is a strange one.

Did you adjust for Zulu time for those slots? The API uses GMT times. [email protected]:00 would be 1600/4pm i think , and [email protected]:00 would be Noon

Another strange thing... I previewed your zip and 23.1 and 23.2 dont show in my preview list but they show in your build list.
 

·
Registered
Joined
·
1,712 Posts
There was a build exception after 12.3, so maybe on your chan 17.1:
LS:BLS: Added 803 programs for channel 12.3, RfChan=13 (StationID=51000)
THRD:BUILDER: Build Schedule Exception occurred!
THRD:SetWDogBuildTimer: Auto Build time set for 2020-10-18 10:05:44(EDT).
THRD:BUILDER: Queue is Empty, releasing SDResourceLock

So it would appear there would be BIG holes on all chans > 12.3

So you could do a 'SaveSettings' build and see if the exception happens again, if so, you could set the log level to debug and 'savesettings' again to see if something else gets caputred. and you have your stdio captured too, right? anything in there?
You're right, I scanned through the guide horizontally tuned to channel 6, so I did not see listings missing for >12.3. But they are missing, except curiously here and there a listing is not unknown, but lists a real program (whether it is correct and for the corresponding time slot I'm not going to research).

So I did a 'SaveSettings" build, and it went perfectly, without an exception. So there is no point in getting a debug log, the moment has passed and I guess we'll never know what caused the exception. Still don't understand the gaps in the requests - does the DVR+ specifically include a date/time slot in its guide data request? If so, it would seem to have skipped a 4-hour slot but retained the old data in that slot. If not, is there any reason you would skip a time slot on consecutive requests?

So there were really two problems logged here, seemingly unrelated, though I wouldn't rule out some strange correlation.

I'll keep an eye on it tomorrow. Can't imagine there is anything on channels >12.3 that we need to record tonight.

N.B.: You'll notice from the log that I had rebooted the Pi yesterday. I have the green led programmed to show CPU activity and normally there is about a 2-second repeating and recognizable blink pattern that reassures me PiGS is idle. When guide is being built, or sent to a DVR+, that cycle is interrupted and replaced by various combinations of solid on, solid off, or a different blink pattern on the green led. But yesterday morning I looked up and saw it was all blinking in a very strange way, so I connected to it via RDP. The desktop came up but it hung after that, not allowing me to switch windows or enter any keystrokes. I waited a long time, and even tried SSH, but nothing would get through, so I rebooted it. After reboot, there were a ton of kernel messages around that time in the messages log file(s), but they were indecipherable to me. So something in the kernel apparently hung. I don't think it was PiGS related. But prior to that PiGS had been up and working fine for about 6 days. But I do like having the green led give me a clue how things are going. I run a shell script for that:
Code:
#!/bin/sh

while true
do
    if ping -c1 192.168.1.1
    then
        # normal
        echo cpu0 | sudo tee /sys/class/leds/led0/trigger
    else
        # abnormal - network connectivity problem
        echo heartbeat | sudo tee /sys/class/leds/led0/trigger
    fi
    sleep 10
done >/dev/null
The 192.168.1.1 is my router. If it can be pinged, CPU activity is sent to the led. If it can't, a heartbeat pattern is sent to the led. AFAIK, this script is only suitable for the Pi Zero W.
 

·
Premium Member
Joined
·
234 Posts
Discussion Starter #620 (Edited)
Except curiously here and there a listing is not unknown, but lists a real program (whether it is correct and for the corresponding time slot I'm not going to research).
Still don't understand the gaps in the requests - does the DVR+ specifically include a date/time slot in its guide data request?
If so, it would seem to have skipped a 4-hour slot but retained the old data in that slot.
Yes, the dvr requests a specific block, shown as GLS:Searching... it also says give me 240 minutes, but I dont print that. It literally appears to have skipped a 4 hr block.

dvr seems to back up to 08:00 local time (12:00Z) for its first request. It seems to want to stay on 00:00 +4:00 blocks

LS:GLS First Request at: 2020-10-16 11:04:27(EDT) <<< Your local time
LS:GLS:Searching for 2020-10-16T12:00:00Z. InProgessFlag=1... (InProg=1 for the first request)
LS:GLS:Searching for 2020-10-16T16:00:00Z. InProgessFlag=0... (This is exactly what the dvr requests... Zulu time)

I guess i mis-read the logs... yesterdays' 16th build completed and both dvrs got 90 blocks.
today the build crashed and a block was skipped. so maybe there was a bad date in the pigs return so dvr decided to advance its request.
that may explain the funny holes

and looking at the code.... it appears 12.3 completes b/c that message prints at the bottom of a loop.
then it starts processing 17.1 (we dont know where it crashes in these next steps)
get 17 days sched data from SD for 17.1
add each prog to a temp list (prog date/time, basic stuff) and build a list of program IDs
get the program details for all program IDs
loop the temp list and add in prog details (titles, description, etc)
add this Temp list to the Master Schedule <<< at this point there is unsorted junk at the end of the master list
Sort the master schedule by time to put the new listing in their proper spots <<< if this were to crash, the master list would be in a partially built state
print the Ch done message <<< message never occurs for 17.1
 
601 - 620 of 631 Posts
Top