Texture Synthesis using Image Quilting

1. Texture Synthesis using the Image Quilting Algorithm From Image Quilting for Texture Synthesis and Transferby Efros & Freeman, SIGGRAPH 2001 Presenter: Vicente Ordonez Department of Computer Science Stony Brook University State University of New York

2. Texture Synthesis Given a sample texture image, generate an output texture of arbitrary size.

3. It Should Work on Different Types of Textures Stochastic textures (no apparent structure)

4. It Should Work on Different Types of Textures Repeated textures (Structured)

5. Other types of textures Semi-structured textures. Repetition and stochastic processes involved.

6. The Taking Patch Samples from the Sample Image Approach Xu et al. from MSR proposed the idea of taking random patch samples from the input image and then blending them together by blurring the edges.

7. Take the Patches so that they Agree on Some Overlapping Region B1 B2 Neighboring blocks constrained by overlap Idea: Select the K patches from the input sample that match closely to the current overlapping region in the output texture. Pick one of those K patches randomly.

8. Algorithm Select a random patch from the input image and place it on the top left corner of the output texture. Go through the output texture in raster scan order with steps of the size of the patch minus the dimension of the overlap region. Using the current overlap region in the output texture, search for every location in the input image that minimizes the sum of square distances error between the overlap with top and left side of the last patch. Choose randomly one of the patches identified in the previous step and paste it into the output texture.

9. How to avoid repeating calculations? Idea: Don’t calculate the SSD for the overlapping regions for each possible positions one-bye-one but do it all in one pass. This is the objective:

10. How to avoid repeating calculations?

11. Blend each new patch along a Minimum Error Boundary Cut Calculate a cut between patches that minimizes the error along the overlap region. B1 B2 Minimal error boundary cut

12. Using Dynamic Programming to Calculate the Minimum Cost Path Generate a minimum cost path table {Eij} using the overlap error area {eij} using Dynamic Programming as follows: Traverse backwards the resulting table to find the minimum cost path.

13. Error Overlap Region and min-cut

14. Running the min-cut algorithm

15. How to get a better blending? Alpha blending Pyramid blending Histogram matching Blending in the gradient domain Poisson Blending

16. User Controlled Parameters The size of the patches. This heavily affects the results. The ideal size patch is the one that captures the minimum units of detail of the objects being repeated in the texture. We also have to be careful not to choose the size patch so large so that the stochastic nature of the results gets affected. The size of the overlap. This mainly affects the cuts between adjacent patches. This parameter also depends on the patterns of the input texture. Having a larger overlap does not always results into a visual improvement in the generated texture. The size of the error range when choosing patches.Making this range larger makes the selection converge into a random selection of patches, so it has to be sufficiently large to take advantage of the overlap constraints while leaving an stochastic process to affect the results.

17. Final Results. Generating some textures. Generated in 12s

18. Final Results. Generating some textures. Generated in 25s

19. Final Results. Generating some textures. Generated in 11 s

20. Final results: Generating some textures. Patch size = 50pxOverlap size = 8pxError range = (0 – 0.05)Execution time = 15sInput texture size = 148x139Output texture size = 428x428

21. Final Results. Generating some textures. Patch size = 20pxOverlap size = 4pxError range = (0 – 0.05)Execution time = 26sInput texture size = 148x138Output texture size = 343x343

22. Conclusions The results of the original paper have been successfully replicated. The execution times are slow for a real time application but there are some hints about how to make it in real time using other research results. Even when there are some recommendations for selecting the parameters, selecting the appropriate ones is an important step for the method and still largely depends on the image contents. Despite the previous fact, the method performs well in a wide range of textures that were difficult to handle for previous methods. Given the appropriate parameters the method seems to resemble well the visual appearance of the input texture.

23. Issues/Further Additions Searching for a patch that agrees on an overlap region is computationally expensive, so it is necessary to search using heuristics or use another kind of efficient search (Tree structures). The blending function still produces a visible seam. Maybe there’s room for improvement here. Pyramid Blending/ Gradient/Poisson Blending? Other ways to choose the seam that allow more flexibility in the curvature of the seam.

24. References Sample textures taken from the original Efros & Freeman paper and the project website athttp://graphics.cs.cmu.edu/people/efros/research/quilting.html