Dialogue

Sketch VQA

• Conducting Sketch VQA to incorporate the context of user-drawn sketches into conversations rather than relying solely on user speech 

• Constructing a Sketch-QA pair dataset and fine-tuning BLIP for VQA on sketches 

Sketch Generation & Inpainting

• Understanding user intent and providing responses through a chatbot built with Rasa, while generating images based on user-drawn sketches and input prompts by combining Stable Diffusion and ControlNet

• Constructing a Sketch Dataset and fine-tuning Stable Diffusion to generate images similar to human-drawn sketches