pkg-perl-maintainers team mailing list archive

Thread
Date
[Bug 1210314] [NEW] get-flash-videos failing to download some BBC news videos (possibly other BBC videos too)

To: pkg-perl-maintainers@xxxxxxxxxxxxxxxxxxx
From: Pigeon <launchpad-ubuntu@xxxxxxxxxxxxxxxxx>
Date: Thu, 08 Aug 2013 22:22:50 -0000
Reply-to: Bug 1210314 <1210314@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

System: Ubuntu 11.04 (but bug is not related to any other packages in system)
Package: get-flash-videos 1.25~git2012.06.27-1

get-flash-videos is failing on certain videos from the BBC, apparently because of a change in the BBC web page code.
Example output from failed run:

$ get_flash_videos --debug http://www.bbc.co.uk/news/uk-23626232
2 plugins installed:
- Hulu.pm
- Itv.pm
Trying to open plugin ~/.get_flash_videos/plugins/Www.pm
Trying to open plugin ~/.get_flash_videos/plugins/Bbc.pm
Using method 'bbc' for http://www.bbc.co.uk/news/uk-23626232
Downloading http://www.bbc.co.uk/news/uk-23626232
-> GET http://www.bbc.co.uk/news/uk-23626232
<- 200 text/html text/html; charset=utf-8 (58275)
Error: Couldn't find BBC XML playlist URL in http://www.bbc.co.uk/news/uk-23626232 at /usr/share/perl5/FlashVideo/Site/Bbc.pm line 60.

Couldn't extract Flash movie URL. This site may need specific support adding,
or fixing.

Please confirm the site is using Flash video and if you have Flash available
check that the URL really works(!).

Check for updates by running: /usr/bin/get_flash_videos --update

If the latest version does not support this please open a bug (or
contribute a patch!) at http://code.google.com/p/get-flash-videos/
make sure you include the output with --debug enabled.
Couldn't download any videos.


The following patch fixes this problem...

$ diff -u /usr/share/perl5/FlashVideo/Site/Bbc.pm.orig /usr/share/perl5/FlashVideo/Site/Bbc.pm
--- /usr/share/perl5/FlashVideo/Site/Bbc.pm.orig        2013-08-08 23:18:52.000000000 +0100
+++ /usr/share/perl5/FlashVideo/Site/Bbc.pm     2013-08-08 22:56:07.000000000 +0100
@@ -18,6 +18,10 @@
   if ($browser->content =~ /<param name="playlist" value="(http:.+?\.s?xml)"/) {
     $playlist_xml = $1;
   }
+  elsif ($browser->content =~ /"href":"(http:[^"]+?playlist\.s?xml)"/) {
+    $playlist_xml = $1;
+    $playlist_xml =~ s/\\//g;
+  }
   elsif($browser->content =~ /empDivReady\s*\(([^)]+)/) {
     my @params = split /,\s*/, $1;


Example output from the same URL with this patch applied:

$ get_flash_videos --debug http://www.bbc.co.uk/news/uk-23626232
2 plugins installed:
- Hulu.pm
- Itv.pm
Trying to open plugin ~/.get_flash_videos/plugins/Www.pm
Trying to open plugin ~/.get_flash_videos/plugins/Bbc.pm
Using method 'bbc' for http://www.bbc.co.uk/news/uk-23626232
Downloading http://www.bbc.co.uk/news/uk-23626232
-> GET http://www.bbc.co.uk/news/uk-23626232
<- 200 text/html text/html; charset=utf-8 (58275)
Playlist URL: http://playlists.bbc.co.uk/news/uk-23626232A/playlist.sxml
-> GET http://playlists.bbc.co.uk/news/uk-23626232A/playlist.sxml
<- 200 text/xml (4864)
-> GET http://open.live.bbc.co.uk/mediaselector/4/mtis/stream/p01dwkgh
<- 200 text/xml (5841)
FlashVideo::RTMPDownloader=HASH(0x9f22610)
Running rtmpdump --verbose --pageUrl 'http://www.bbc.co.uk/news/uk-23626232' --app 'ondemand?_fcs_vhost=cp45413.edgefcs.net&auth=daEdWclbWbVb4dTc4bmckdtdId9bFaIbzdC-bsbbn8-bWG-JpmEIoDoLEsGtxE&aifp=v001&slist=public/mps_h264_200/public/news/uk/1001000/1001225_h264_176k.mp4;public/mps_h264_400/public/news/uk/1001000/1001225_h264_512k.mp4;public/mps_h264_med/public/news/uk/1001000/1001225_h264_800k.mp4;public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4' --tcUrl 'rtmp://cp45413.edgefcs.net/ondemand?_fcs_vhost=cp45413.edgefcs.net&auth=daEdWclbWbVb4dTc4bmckdtdId9bFaIbzdC-bsbbn8-bWG-JpmEIoDoLEsGtxE&aifp=v001&slist=public/mps_h264_200/public/news/uk/1001000/1001225_h264_176k.mp4;public/mps_h264_400/public/news/uk/1001000/1001225_h264_512k.mp4;public/mps_h264_med/public/news/uk/1001000/1001225_h264_800k.mp4;public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4' --swfUrl 'http://news.bbc.co.uk/player/emp/2.11.7978_8433/9player.swf' --rtmp 'rtmp://cp45413.edgefcs.net/ondemand' --flv 'BBC_-_Co-op_supermarket_to_stop_selling_Nuts_magazine.flv' --playpath 'mp4:public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4?auth=daEdWclbWbVb4dTc4bmckdtdId9bFaIbzdC-bsbbn8-bWG-JpmEIoDoLEsGtxE&aifp=v001&slist=public/mps_h264_200/public/news/uk/1001000/1001225_h264_176k.mp4;public/mps_h264_400/public/news/uk/1001000/1001225_h264_512k.mp4;public/mps_h264_med/public/news/uk/1001000/1001225_h264_800k.mp4;public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4'
RTMPDump v2.4
(c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
rtmpdump: DEBUG: Parsing...
rtmpdump: DEBUG: Parsed protocol: 0
rtmpdump: DEBUG: Parsed host    : cp45413.edgefcs.net
rtmpdump: DEBUG: Parsed app     : ondemand
rtmpdump: DEBUG: Protocol : RTMP
rtmpdump: DEBUG: Hostname : cp45413.edgefcs.net
rtmpdump: DEBUG: Port     : 1935
rtmpdump: DEBUG: Playpath : mp4:public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4?auth=daEdWclbWbVb4dTc4bmckdtdId9bFaIbzdC-bsbbn8-bWG-JpmEIoDoLEsGtxE&aifp=v001&slist=public/mps_h264_200/public/news/uk/1001000/1001225_h264_176k.mp4;public/mps_h264_400/public/news/uk/1001000/1001225_h264_512k.mp4;public/mps_h264_med/public/news/uk/1001000/1001225_h264_800k.mp4;public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4
rtmpdump: DEBUG: tcUrl    : rtmp://cp45413.edgefcs.net/ondemand?_fcs_vhost=cp45413.edgefcs.net&auth=daEdWclbWbVb4dTc4bmckdtdId9bFaIbzdC-bsbbn8-bWG-JpmEIoDoLEsGtxE&aifp=v001&slist=public/mps_h264_200/public/news/uk/1001000/1001225_h264_176k.mp4;public/mps_h264_400/public/news/uk/1001000/1001225_h264_512k.mp4;public/mps_h264_med/public/news/uk/1001000/1001225_h264_800k.mp4;public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4
rtmpdump: DEBUG: swfUrl   : http://news.bbc.co.uk/player/emp/2.11.7978_8433/9player.swf
rtmpdump: DEBUG: pageUrl  : http://www.bbc.co.uk/news/uk-23626232
rtmpdump: DEBUG: app      : ondemand?_fcs_vhost=cp45413.edgefcs.net&auth=daEdWclbWbVb4dTc4bmckdtdId9bFaIbzdC-bsbbn8-bWG-JpmEIoDoLEsGtxE&aifp=v001&slist=public/mps_h264_200/public/news/uk/1001000/1001225_h264_176k.mp4;public/mps_h264_400/public/news/uk/1001000/1001225_h264_512k.mp4;public/mps_h264_med/public/news/uk/1001000/1001225_h264_800k.mp4;public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4
rtmpdump: DEBUG: live     : no
rtmpdump: DEBUG: timeout  : 30 sec
rtmpdump: DEBUG: Setting buffer time to: 36000000ms
Connecting ...
rtmpdump: DEBUG: RTMP_Connect1, ... connected, handshaking
rtmpdump: DEBUG: HandShake: Type Answer   : 03
rtmpdump: DEBUG: HandShake: Server Uptime : 739889075
rtmpdump: DEBUG: HandShake: Handshaking finished....
rtmpdump: DEBUG: HandleServerBW: server BW = 1250000
chunk size change to 4096
DEBUG: RTMP_ClientPacket, received: invoke 240 bytes
DEBUG: (object begin)
DEBUG: (object begin)
DEBUG: Property: <Name:             fmsVer, STRING:     FMS/4,5,2,524>
DEBUG: Property: <Name:       capabilities, NUMBER:     127.00>
DEBUG: Property: <Name:               mode, NUMBER:     1.00>
DEBUG: (object end)
DEBUG: (object begin)
DEBUG: Property: <Name:              level, STRING:     status>
DEBUG: Property: <Name:               code, STRING:     NetConnection.Connect.Success>
DEBUG: Property: <Name:        description, STRING:     Connection succeeded.>
DEBUG: Property: <Name:     objectEncoding, NUMBER:     0.00>
DEBUG: Property: <Name:               data, OBJECT>
DEBUG: (object begin)
DEBUG: Property: <Name:            version, STRING:     4,5,2,524>
DEBUG: (object end)
DEBUG: (object end)
DEBUG: (object end)
DEBUG: HandleInvoke, server invoking <_result>
DEBUG: HandleInvoke, received result for method call <connect>
DEBUG: sending ctrl. type: 0x0003
DEBUG: Invoking createStream
DEBUG: RTMP_ClientPacket, received: invoke 21 bytes
DEBUG: (object begin)
DEBUG: Property: NULL
DEBUG: (object end)
DEBUG: HandleInvoke, server invoking <onBWDone>
DEBUG: Invoking _checkbw
rtmpdump: DEBUG: RTMP_ClientPacket, received: invoke 29 bytes
eInvoke, server invoking <_result>
DEBUG: HandleInvoke, received result for method call <createStream>
DEBUG: SendPlay, seekTime=0, stopTime=0, sending play: mp4:public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4?auth=daEdWclbWbVb4dTc4bmckdtdId9bFaIbzdC-bsbbn8-bWG-JpmEIoDoLEsGtxE&aifp=v001&slist=public/mps_h264_200/public/news/uk/1001000/1001225_h264_176k.mp4;public/mps_h264_400/public/news/uk/1001000/1001225_h264_512k.mp4;public/mps_h264_med/public/news/uk/1001000/1001225_h264_800k.mp4;public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4
DEBUG: Invoking play
DEBUG: sending ctrl. type: 0x0003
rtmpdump: DEBUG: RTMP_ClientPacket, received: invoke 10275 bytes
ndleInvoke, server invoking <_onbwcheck>
DEBUG: Invoking _result
rtmpdump: DEBUG: HandleCtrl, received ctrl. type: 4, len: 6
rtmpdump: DEBUG: HandleChangeChunkSize, received: chunk size change to 4096
eCtrl, Stream Begin 1
DEBUG: RTMP_ClientPacket, received: invoke 270 bytes
DEBUG: (object begin)
DEBUG: Property: NULL
DEBUG: (object begin)
DEBUG: Property: <Name:              level, STRING:     status>
DEBUG: Property: <Name:               code, STRING:     NetStream.Play.Reset>
DEBUG: Property: <Name:        description, STRING:     Playing and resetting public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4.>
DEBUG: Property: <Name:            details, STRING:     public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4>
DEBUG: Property: <Name:           clientid, STRING:     qAAOwpJA>
DEBUG: (object end)
DEBUG: (object end)
DEBUG: HandleInvoke, server invoking <onStatus>
DEBUG: HandleInvoke, onStatus: NetStream.Play.Reset
rtmpdump: DEBUG: RTMP_ClientPacket, received: invoke 264 bytes
operty: <Name:              level, STRING:      status>
DEBUG: Property: <Name:               code, STRING:     NetStream.Play.Start>
DEBUG: Property: <Name:        description, STRING:     Started playing public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4.>
DEBUG: Property: <Name:            details, STRING:     public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4>
DEBUG: Property: <Name:           clientid, STRING:     qAAOwpJA>
DEBUG: (object end)
DEBUG: (object end)
DEBUG: HandleInvoke, server invoking <onStatus>
DEBUG: HandleInvoke, onStatus: NetStream.Play.Start
Starting download at: 0.000 kB
DEBUG: RTMP_ClientPacket, received: notify 24 bytes
DEBUG: (object begin)
DEBUG: (object end)
DEBUG: ignoring too small audio packet: size: 0
DEBUG: RTMP_ClientPacket, received: notify 44 bytes
DEBUG: (object begin)
DEBUG: (object begin)
DEBUG: Property: <Name:               code, STRING:     NetStream.Data.Start>
DEBUG: (object end)
DEBUG: (object end)
rtmpdump: DEBUG: RTMP_ClientPacket, received: notify 494 bytes
   duration, NUMBER:    123.24>
DEBUG: Property: <Name:       moovPosition, NUMBER:     36.00>
DEBUG: Property: <Name:              width, NUMBER:     704.00>
DEBUG: Property: <Name:             height, NUMBER:     396.00>
DEBUG: Property: <Name:       videocodecid, STRING:     avc1>
DEBUG: Property: <Name:       audiocodecid, STRING:     mp4a>
DEBUG: Property: <Name:         avcprofile, NUMBER:     77.00>
DEBUG: Property: <Name:           avclevel, NUMBER:     30.00>
DEBUG: Property: <Name:             aacaot, NUMBER:     2.00>
DEBUG: Property: <Name:     videoframerate, NUMBER:     25.00>
DEBUG: Property: <Name:    audiosamplerate, NUMBER:     44100.00>
DEBUG: Property: <Name:      audiochannels, NUMBER:     1.00>
DEBUG: Property: <Name:          trackinfo, OBJECT>
DEBUG: (object begin)
DEBUG: (object begin)
DEBUG: Property: <Name:             length, NUMBER:     3081000.00>
DEBUG: Property: <Name:          timescale, NUMBER:     25000.00>
DEBUG: Property: <Name:           language, STRING:     und>
DEBUG: Property: <Name:  sampledescription, OBJECT>
DEBUG: (object begin)
DEBUG: (object begin)
DEBUG: Property: <Name:         sampletype, STRING:     avc1>
DEBUG: (object end)
DEBUG: (object end)
rtmpdump: DEBUG: (object end)
:          timescale, NUMBER:   44100.00>
DEBUG: Property: <Name:           language, STRING:     und>
DEBUG: Property: <Name:  sampledescription, OBJECT>
DEBUG: (object begin)
DEBUG: (object begin)
DEBUG: Property: <Name:         sampletype, STRING:     mp4a>
DEBUG: (object end)
DEBUG: (object end)
DEBUG: (object end)
DEBUG: (object end)
DEBUG: (object end)
DEBUG: (object end)
INFO: Metadata:
INFO:   duration              123.24
INFO:   moovPosition          36.00
INFO:   width                 704.00
INFO:   height                396.00
INFO:   videocodecid          avc1
INFO:   audiocodecid          mp4a
INFO:   avcprofile            77.00
INFO:   avclevel              30.00
INFO:   aacaot                2.00
INFO:   videoframerate        25.00
INFO:   audiosamplerate       44100.00
INFO:   audiochannels         1.00
INFO: trackinfo:
INFO:   length                3081000.00
INFO:   timescale             25000.00
INFO:   language              und
INFO: sampledescription:
INFO:   sampletype            avc1
INFO:   length                5434368.00
INFO:   timescale             44100.00
INFO:   language              und
INFO: sampledescription:
INFO:   sampletype            mp4a
DEBUG: RTMP_ClientPacket, received: invoke 57 bytesne.flv: 0% (53.28 / 53277.00 KiB)
DEBUG: (object begin)
DEBUG: Property: NULL
DEBUG: (object end)
DEBUG: HandleInvoke, server invoking <onBWDone>
rtmpdump: DEBUG: RTMP_ClientPacket, received: invoke 40 bytes
eInvoke, server invoking <_onbwdone>
BBC_-_Co-op_supermarket_to_stop_selling_Nuts_magazine.flv: 99% (22530.67 / 22712.37 KiB)rtmpdump: DEBUG: ignoring too small audio packet: size: 0
rtmpdump: DEBUG: RTMP_ClientPacket, received: notify 102 bytes
rtmpdump: DEBUG: (object begin)
rtmpdump: DEBUG: (object begin)
rtmpdump: DEBUG: Property: <Name:               code, STRING:   NetStream.Play.Complete>
rtmpdump: DEBUG: Property: <Name:              level, STRING:   status>
rtmpdump: DEBUG: Property: <Name:           duration, NUMBER:   103.00>
rtmpdump: DEBUG: Property: <Name:              bytes, NUMBER:   23271822.00>
rtmpdump: DEBUG: (object end)
rtmpdump: DEBUG: (object end)
rtmpdump: DEBUG: HandleCtrl, received ctrl. type: 1, len: 6
rtmpdump: DEBUG: HandleCtrl, Stream EOF 1
rtmpdump: DEBUG: RTMP_ClientPacket, received: invoke 274 bytes
rtmpdump: DEBUG: (object begin)
rtmpdump: DEBUG: Property: NULL
rtmpdump: DEBUG: (object begin)
rtmpdump: DEBUG: Property: <Name:              level, STRING:   status>
rtmpdump: DEBUG: Property: <Name:               code, STRING:   NetStream.Play.Stop>
rtmpdump: DEBUG: Property: <Name:        description, STRING:   Stopped playing public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4.>
rtmpdump: DEBUG: Property: <Name:            details, STRING:   public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4>
rtmpdump: DEBUG: Property: <Name:           clientid, STRING:   qAAOwpJA>
rtmpdump: DEBUG: Property: <Name:             reason, STRING:   >
rtmpdump: DEBUG: (object end)
rtmpdump: DEBUG: (object end)
rtmpdump: DEBUG: HandleInvoke, server invoking <onStatus>
rtmpdump: DEBUG: HandleInvoke, onStatus: NetStream.Play.Stop
rtmpdump: DEBUG: Invoking deleteStream
rtmpdump: DEBUG: Got Play.Complete or Play.Stop from server. Assuming stream is complete
rtmpdump: DEBUG: RTMP_Read returned: 3655
rtmpdump: DEBUG: Closing connection.

Done. Saved 23271668 bytes to BBC_-_Co-
op_supermarket_to_stop_selling_Nuts_magazine.flv

** Affects: get-flash-videos (Ubuntu)
     Importance: Undecided
         Status: New

** Patch added: "Patch to fix bug downloading BBC videos"
   https://bugs.launchpad.net/bugs/1210314/+attachment/3765415/+files/get-flash-videos-bbc-patch

-- 
You received this bug notification because you are a member of Debian
Perl Group, which is subscribed to get-flash-videos in Ubuntu.
https://bugs.launchpad.net/bugs/1210314

Title:
  get-flash-videos failing to download some BBC news videos (possibly
  other BBC videos too)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/get-flash-videos/+bug/1210314/+subscriptions
Follow ups

[Bug 1210314] Re: get-flash-videos failing to download some BBC news videos (possibly other BBC videos too)
From: wonderfullifeness, 2015-10-19
[Bug 1210314] Re: get-flash-videos failing to download some BBC news videos (possibly other BBC videos too)
From: Luite van Zelst, 2013-09-01
[Bug 1210314] Re: get-flash-videos failing to download some BBC news videos (possibly other BBC videos too)
From: Brian Murray, 2013-08-12
[Bug 1210314] Re: get-flash-videos failing to download some BBC news videos (possibly other BBC videos too)
From: Pigeon, 2013-08-09
[Bug 1210314] Re: get-flash-videos failing to download some BBC news videos (possibly other BBC videos too)
From: Brian Murray, 2013-08-09
[Bug 1210314] Re: get-flash-videos failing to download some BBC news videos (possibly other BBC videos too)
From: Pigeon, 2013-08-09
[Bug 1210314] Re: get-flash-videos failing to download some BBC news videos (possibly other BBC videos too)
From: Ubuntu Foundations Team Bug Bot, 2013-08-09
[Bug 1210314] [NEW] get-flash-videos failing to download some BBC news videos (possibly other BBC videos too)
From: Pigeon, 2013-08-08
References

[Bug 1210314] [NEW] get-flash-videos failing to download some BBC news videos (possibly other BBC videos too)
From: Pigeon, 2013-08-08