This patent is subject to an expiration for failure to pay maintenance fees. See comments below.
Real-time method of digitally altering a video data stream to remove portions of the original image and substitute elements to create a new image
Nickname - "The alleged iChat patent"
A method that allows the real-time replacement of the designated background portion of an incoming video signal with an alternate background. The method utilizes the actual background image for reference as the basis for determining the background and foreground elements within the image with the end result being comparable to traditional blue-screen processes, such as chroma-key and ultimate technology, but requiring only a personal computer, video camera and the software. In this case, however, the reference background image can be any reasonably static scene with a sufficient and stable light source captured by the camera. The video data stream is modified in real-time by comparisons against the reference background image and is then passed on to its original destination. Multiple signal-noise processing algorithms are applied in real-time against the signal to achieve a visually acceptable matte.
GotRenewables? - Track the latest developments in renewable energy
1. A method of digitally altering a video data stream representing an original image in real-time, using a computer system having a memory, and a visual display screen, to
remove portions of the original video image and substitute new elements to create a new video image comprising the steps of:
storing at least one frame of data representing said original image in a first incoming video data stream in a data structure in a first location in said memory in a computer;
displaying said stored original image on said visual display;
capturing real-time video from a second incoming video data stream representing a subsequently modified original image and storing data representing said modified original image in a data structure in a second location in said memory in a
computer;
comparing video data in the second video data stream representing said original image with modifications with said stored original image video data to differentiate the background and foreground elements;
removing the common background elements of said stored original image and the subsequently modified original image leaving only foreground elements of said subsequently modified original image;
replacing said background elements of said subsequently modified original image with alternate background replacement elements; and
displaying only the foreground elements in said subsequently modified original image upon the said alternate background replacement elements to form said new image.
2. The method as in claim 1 further comprising the steps of:
using a stationary video camera to obtain said original image in the form of said first incoming video data and said subsequently modified original image in the form of second incoming video data;
coupling said camera first and second incoming video data to said computer; and
digitizing said first incoming video data of said original image and said second incoming video data of said second real-time video data stream for storage in respective first and second locations in said memory.
3. The method as in claim 2 further comprising the steps of:
causing said original image to be reasonably free of movement; and
lighting said original image sufficiently to enable the detection and separation of background and foreground elements when said digitized second real-time incoming video data stream is compared with said stored digitized original image.
4. The method as in claim 3 further comprising the steps of:
creating said first and second video signals with pixel receptors in said video camera, each pixel receptor generating an output signal containing values that represent the stored original image carried by the first incoming video data stream and
the stored real-time incoming original image carried by the second video data stream; and
creating a first set of data structures in said memory for storing values from said pixel receptors that represent variations in the original image, variations in the dynamics of the original image lighting, and variations in pixel signal values
caused by the camera pixel receptors generating the first incoming video data stream representing said original image.
5. The method as in claim 4 wherein the step of comparing the video data in said second real-time video data stream representing said modified original image with the stored original image data further comprises the steps of:
creating a second set of data structures in said memory for storing data that represents variations in the second real-time original image, variations in the dynamics of the lighting, and the signal variations in pixel signal values caused by the
camera pixel receptors for the second real-time original image; and
comparing pixel values of said data stored in said second data structure for said second real-time original image with corresponding pixel data values stored in said first data structure to determine foreground and background elements in said
second real-time original image.
6. The method as in claim 5 further comprising the steps of: comparing the data in said second real-time video data stream representing said modified original image with the stored original image data; and
generating signals representing only said foreground elements for display on said alternate background replacement elements.
7. The method as in claim 6 further including the step of adding a person to said second real-time video data stream to replace at least a portion of said original image.
8. The method as in claim 6 wherein the step of forming said alternate replacement image further comprises the step of providing one of a moving video, a bitmap, or animation, or any image capable of being represented in a digital format, as the alternate replacement image.
9. A computer-aided system for digitally editing, in real-time, a video data stream representing an original image by identifying and separating portions of the original image into foreground and background elements and replacing the background
elements of the original image with substitute elements to create a new image, the system comprising:
a visual display for displaying said original image contained in said video data stream;
a camera having a device associated therewith for digitizing video data representing said original image and captured by said camera;
a computer coupled to said visual display and said digitizing device;
a first memory in the computer for storing the digitized original image from said video data stream;
said camera and digitizing device capturing and digitizing a second real-time video data stream containing data representing the original image;
a second memory in the computer for storing the digitized second real-time original image from said video data stream;
a third memory in the computer for storing program instructions for comparing the digitized second real-time original image with the stored original image to detect and separate foreground and background elements;
an alternate replacement background image stored at a fourth memory available to said stored program instructions for display on said visual display;
said second real-time original image being modified by the background elements thereof being replaced with the alternate replacement background image;
said camera and associated device capturing and digitizing said modified original image having the replacement background image; and
said program instructions comparing said modified original image with said stored original image to obtain only foreground elements and causing said foreground elements to be displayed upon the alternate replacement image to form the new image.
10. A computer memory product containing a program for causing real-time digital alteration of a video data stream from pixel receptors in a video camera that represents an original image by removing portions of the original image and
substituting elements to create a new image, the program comprising the steps of:
causing video data in a first video data stream representing the original image to be stored in a first computer memory location;
causing a comparison of a real-time video data stream representing the original image having substitute portions added thereto with the stored original image to obtain only the substitute portions; and
controlling the computer in response to program instructions stored in the memory of the computer to cause a visual display of only the substitute portions of the original image upon an alternate replacement image to form a new image.
11. The program of claim 10 further comprising the steps of:
accessing a first set of data structures in the first memory location in the computer that stores digitized video signals that represent the original image carried by the first video data stream including variations in the dynamics of the
original image lighting, motion, and variations in pixel signal values caused by the video camera pixel receptors;
accessing a second set of data structures in the computer memory that stores data that represents variations of the original image with substitute portions therein in a second real-time video data stream, variations in the dynamics of the
lighting of the original image, motion, and the signal variations in pixel signal values for the second real time video data stream; and
controlling the computer to cause a comparison of the stored data from the second real-time video stream, including substitute portions, with the stored original image data to determine foreground and background elements .
Comments from the community
7 comments have been received for United States Patent No. 5,764,306. Want to add another?
#1|FizzMaster comments:
United States Patent No. 5,764,306 expired on June 9, 2006, due to a failure to pay applicable maintenance fees. See the Official Gazette notice here.
For more information on patent maintenance fees, see 35 U.S.C. 41(b) and 37 C.F.R. 1.362(g).
#2|FizzMaster comments:
United States Patent No. 5,764,306 was reinstated on June 13, 2007 following an expiration due to a failure to pay applicable maintenance fees. See the Official Gazette Notice from the United States Patent and Trademark Office here.
For more information on reinstatement of expired patents, see 35 U.S.C. 41(c) and 37 C.F.R. 1.378.
Patent 5,764,306 expired on June 9, 2006 (see the Official Gazette Notice here). The patent was expired for a period of 12 months and 4 days.
#3|Anonymous comments:
Digital Background Corporation has asserted this patent against Apple, claiming that iChat 4.0 infringes the patent. This latest version of iChat software shipped in October, 2007, as part of the latest update to the Mac OS X operating system (Leopard).
#4|Anonymous comments:
Wow...the timing on this one could really test 35 U.S.C. 41(c)(2). Patent was expired from summer '06 to summer '07. Leopard (and iChat 4.0) was released in October '07, making it likely that the program (and the video backdrop feature) was being worked on during that time frame.
Any chance the Apple knew about the expiration - and indeed relied upon it as the new iChat was taking shape?
This one's going to be fun to watch.
#5|FizzMaster comments:
TrollTracker has a thorough write-up on the filing of the suit against Apple.
#6|Appler comments:
The '306 patent and the lawsuit against Apple is being discussed at length in the AppleInsider forums. Click here.
#7|Appler comments:
Patent 5,428,401 appears relevant, and was not cited by the Examiner during prosecution of the '306 patent.
Here's the abstract:
"An image processing system operates on first and second image data (V and Bg) to produce data defining a combined image. The first image data (V) represents a keyed image comprising a first object (F) keyed over a portion of a first background (B) in accordance with keying data (K). The second image data (Bg) represents a background. The system is arranged so that the second image data (Bg) is weighted by respective control data (.alpha.) related to said keying data (K) before being combined with the first image data (V). In another embodiment the first image data represents a second object independently keyed over the keyed image in accordance with second keying data and the control data is further related to the second keying data. In a further embodiment the first image data represents an image derived by cross fading between the first keyed image and a second keyed image."
Indeed, several patents assigned to Quantel Limited of the UK appear relevant. View a complete listing of their portfolio here.