Generating bounding boxes with UDOP #87

Activity

1. zinengtang commented on Jul 17, 2023

   

   Zineng Tang (zinengtang)

   on Jul 17, 2023

   Collaborator

   More actions

   We have some similar objectives. For example, question answering, the answer will be followed by its bounding box. So, this is possible indeed as long as the format follows "[text sequence] "
2. AleRosae commented on Jul 18, 2023

   

   AleRosae

   on Jul 18, 2023

   Author

   More actions

   Thank you for your answer Zineng Tang (@zinengtang)!
   So if I'm not mistaking, to do so we should first normalize the original bounding boxes in range [0, 1000] on the basis of width and height of the original image; then normalize them between [0, 1]; and then convert them into layout tokens by multiplying them for the layout vocabulary size (500). Am I getting it right?

   Btw, I'm using the (not yet merged) code from the HuggingFace PR that is porting UDOP into Transformers. Works like a charm, but there might be some differences with your code.
3. sromoam commented on Jul 18, 2023

   

   Spencer Romo (sromoam)

   on Jul 18, 2023

   More actions

   AleRosae can you share any snippets of your use of the PR? I got stuck on an early step.

   Thanks in advance.
4. AleRosae commented on Jul 19, 2023

   

   AleRosae

   on Jul 19, 2023

   Author

   More actions

   Hi Spencer Romo (@sromoam),
   for inference you can use the standard generate() method:

   model = UdopForConditionalGeneration.from_pretrained("udop_model")
   outputs = model.generate(input_ids=input_ids,
                                         bbox=bbox,
                                         attention_mask=attention_mask,
                                         pixel_values=pixel_values,
                                      max_length=512,
                                      use_cache=False,
                                      num_beams=1,
                                      return_dict_in_generate=True)
   

   You can obtain input_ids, bboxes, attention_mask and pixel values using the UdopProcessor:

   processor = UdopProcessor.from_pretrained("udop_model", apply_ocr=True)
   encoding = processor(images=image, return_tensors="pt").to(device)
   

   For finetuning, you can follow the Pix2Struct tutorial. Just be sure to also include words and bboxes in your dataloader, as Pix2Struct only takes images as input.
5. jainamhdoshi commented on Mar 19, 2024

   

   Jainam Doshi (jainamhdoshi)

   on Mar 19, 2024

   More actions

   Hi Spencer Romo (@sromoam), for inference you can use the standard generate() method:

   model = UdopForConditionalGeneration.from_pretrained("udop_model")
   outputs = model.generate(input_ids=input_ids,
                                         bbox=bbox,
                                         attention_mask=attention_mask,
                                         pixel_values=pixel_values,
                                      max_length=512,
                                      use_cache=False,
                                      num_beams=1,
                                      return_dict_in_generate=True)
   

   You can obtain input_ids, bboxes, attention_mask and pixel values using the UdopProcessor:

   processor = UdopProcessor.from_pretrained("udop_model", apply_ocr=True)
   encoding = processor(images=image, return_tensors="pt").to(device)
   

   For finetuning, you can follow the Pix2Struct tutorial. Just be sure to also include words and bboxes in your dataloader, as Pix2Struct only takes images as input.

   can you please provide which libraries did you import for UdopForConditionalGeneration as i am getting error like this
   error : ImportError: cannot import name 'UdopForConditionalGeneration' from 'transformers'
6. jainamhdoshi commented on Mar 19, 2024

   

   Jainam Doshi (jainamhdoshi)

   on Mar 19, 2024

   More actions

   Hi Spencer Romo (@sromoam), for inference you can use the standard generate() method:

   model = UdopForConditionalGeneration.from_pretrained("udop_model")
   outputs = model.generate(input_ids=input_ids,
                                         bbox=bbox,
                                         attention_mask=attention_mask,
                                         pixel_values=pixel_values,
                                      max_length=512,
                                      use_cache=False,
                                      num_beams=1,
                                      return_dict_in_generate=True)
   

   You can obtain input_ids, bboxes, attention_mask and pixel values using the UdopProcessor:

   processor = UdopProcessor.from_pretrained("udop_model", apply_ocr=True)
   encoding = processor(images=image, return_tensors="pt").to(device)
   

   For finetuning, you can follow the Pix2Struct tutorial. Just be sure to also include words and bboxes in your dataloader, as Pix2Struct only takes images as input.

   can you please provide which libraries did you import for UdopForConditionalGeneration as i am getting error like this error : ImportError: cannot import name 'UdopForConditionalGeneration' from 'transformers'

   solved the issue we need to have transformers version = "4.39.0.dev0"
   which can be cloned from here https://github.com/huggingface/transformers/blob/main/src/transformers/__init__.py
   the commit on Mar 18, 2024
7. Joao-M-Silva commented on Jun 7, 2024

   

   João Miguel Freitas da Silva (Joao-M-Silva)

   on Jun 7, 2024

   Last edited by Joao-M-Silva

   More actions

   Zineng Tang (@zinengtang) I want to use the processor with my own OCR. What should be the format of the bouding boxes? 1. Normalized with heigh and width? 2. Normalized with height and width * 1000 3. Other option?