video - Using FFMPEG: How to do a Scene Change Detection? with timecode?

Question

Based on this article it seems that it is possible to use FFMPEG to detect scene change in videos: http://www.luckydinosaur.com/u/ffmpeg-scene-change-detector

Now I have a video that displays a book text and when the text (word or sentence) is spoken it gets highlighted. Something like this audio book: https://youtu.be/lA7L6ZNVKjc

I need to know the timestamp when the text gets highlighted (hence scene change), this will allow me to add timestamp tags on my youtube video, so it becomes easier for listeners to navigate through the audiobook.

What is the magic command line that would do this?

Thank you very much!

Answer 1

Combining the scene filter (for detecting scene changes) and the showinfo filter should achieve what you want:

ffmpeg -i input.flv  
       -filter:v "select='gt(scene,0.4)',showinfo" 
       -f null 
       - 2> ffout

This command extracts all frames that differ from the previous frame by more than (gt) 0.4 (on a scale from 0 to 1). For these frames, information is printed out (showinfo) like this

[Parsed_showinfo_1 @ 0x2d85e60] n:   0 pts:2537204 pts_time:2.5372  pos:  2998114 fmt:rgb24 sar:1/1 s:1920x1200 i:P iskey:1 type:I checksum:5616582E plane_checksum:[5616582E]

Now you only have to extract the timestamp. I think you're interested in pts_time. You could do it like this:

grep showinfo ffout | grep pts_time:[0-9.]* -o | grep [0-9.]* -o > timestamps

This will give you the list of all timestamps:

2.5372
4.37799
6.65301
8.09344

For this approach to work, you must have a version of FFmpeg that implements the scene detection. Also, you have to select a suitable value for the threshold (0.4 in the first command). You can try to find an optimal threshold by extracting frames for different thresholds (and afterwards examine the frames manually) like this

ffmpeg -i input.flv 
       -filter:v "select='gt(scene,0.1)',showinfo" 
       -vsync 0 frames/%05d.jpg

Just for clarification: grep [0-9.]* does not exclude integers as claimed in another answer. It matches any character sequence consisting of digits and periods but it would also match non-numbers like '4.4.4'. However, ffmpeg shouldn't output such ill-formed timestamps.