Skip to content
This repository was archived by the owner on Dec 16, 2022. It is now read-only.

Commit e47da99

Browse files
jacob-morrisondirkgr
jacob-morrison
and
dirkgr
authored
Flickr30k (#285)
* max instances for debugging * b * printing devices * moving tensors? * self * p * p * l * l * fixing heap? * stop logging and printing * less prints * printing devices * p * . * devices * device * test * testing not sampling * testing not using model again * test not moving tensors * not printing * trying image subset * debugging model * going back to full (model is slow?) * right number of instances * distribut * more potential hard negatives * non-distributed * distributed + adding another seen set * fixing evaluation method * format * testing fixed eval * fixing variable * fixing training var * testing new eval again * fix * fix * fix? * changing k to 5 * float * moving labels to gpu * long * trying hopefully fixed loss function * fix * testing out the whole thing * setting max instances to debug in distributed * debug stuff * fixing num images * hopefully fixing the dataset reader in dist * full data * testing out brand new changes * deleting some old comments * fixing validation bug * testing on 1 gpu for now * feature cache broken? * switching to tensor fields and stuff * fix * print device * trying to not move the batch? * moving small batches to cpu * not printing device * deleting old tensor? * debug * printing memory allocation * moving tensor to cpu immediately? * deleting batch? * debug * debug * debug * does this work? * switching to eval and no grad * fix * mask list * backbone roll * typo * log * debug * notes * testing no grad * testing validation batch size of 1 * bug * didn't have the right variable? * don't need to softmax? * trying flickr30k with 8 batch and dummy captions * full flickr? * batch size 1 * testing training batches for validation * Testing out val stuff * updating reader (test will fail for now) * debug statements to figure out why val isn't worki * testing if top images always have same scores * getting rid of caption debugging step * using the right caption var * updating reader to mirror vilbert training setup * full dataset (dummy caption embeddings) * switching to real caption embeddings * testing caching hard negatives * log * limit instances to test caching * delete faiss * more cache tests * one more log statement * single epoch to calculate hard negatives * need to import logging * don't log misses anymore (too slow) * using consistent hash function (test # instances) * Flickr30k batching (#277) merge main + caching captions * test caching captions and hard negatives on full * don't log cache hits * logging training labels to debug * switching val to 4 way mc * can we overfit * not 1k instances * not logging + overfit * not overfit * even fewer instances * all instances * even more overfitting * back to normal * b * bkac to normal * log loss and stuff again * reset * don't include hard negatives in case there's a bug * batch size of 1 * more epochs * only correct answer and hard negatives * Cleanup * Fix error in caption caching * Find hard negatives even when we don't have enough instances * O(1) algorithm for finding a random number with one exception * Make sure the wrong caption comes from a different image * Cross entropy loss * trying overfitting with full instances * use full dataset without learning rate scheduler * don't limit instances and don't log * batch size, scheduler, wandb * comment out wandb * full dataset no hard negatives * don't log loss * giving the correct answer a cheat word * use local feature cache * logging cache stuff * different local feature cache dir * switching to cheat box * bug * something up with some boxes * no cheating and no hard negatives * seeing is a really big batch size works * bug * testing 64 bs * batch size 32 * batch size 48 * full training with 32 batch size no hard negatives * more gradient accumulation steps * trying to train with 10% of the data * fix * bumping up the learning rate, don't correct bias * gradient accumulation + hard negatives * use local feature cache * changing params back * trying real validation * no hard negatives * hard negatives and not real validation * no hard negatives + real validation * calc hn * fixing predictors * fix * fix * fix * fix * cleaning up PR (in progress) * cleaning things up * more cleanup * change warmup steps * only validate every ~5 epochs * printing shapes * more logging * fix log * try cat instead of stack * different logging * test * fix * try batches per epoch * bug * get rid of log statement * use local feature cache * log * logging cache miss * switching back to old captions to use cache * switching back to preprocesing captions * using nfs * Disabling hard negatives to test epoch strat * not logging cache misses * write to local cache (faster) * epoch multiplier * no hard negatives * hard negatives * lowering number of warmup steps * no hard negatives * hard negatives * no hard negatives * hard negatives * Trying Jiasen's featurizer (1x epoch mult) * null image stuff * null image * don't featurize captions (no hn) * adding vilbert ir model tests * cleanup + test distributed * cleanup + dist * test distributed * don't use shard_iterable * fix feature dir * changelog * reformat * log shapes * removing unused vars * using old features * style * lint * lint * don't log shapes * lint * fixing type * debug * changing test files to hopefully fix test * using cloud link for data dir * cleanup * delete print * comment * cleanup * fixing test assert * committing a bunch of fixes * not distributed * fixing metrics * Adding test files + upping max instances * fixes * Switching back to nfs cache * renaming n * update comment * fix * making test deterministic? * sorting files to hopefully achieve consistency Co-authored-by: Dirk Groeneveld <dirkg@allenai.org>
1 parent fb35b2d commit e47da99

32 files changed

Lines changed: 1178 additions & 2 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1313
- Added `StanfordSentimentTreeBankDatasetReader.apply_token_indexers()` to add token_indexers rather than in `text_to_instance`
1414
- Added `AdversarialBiasMitigator` tests.
1515
- Added `adversarial-binary-gender-bias-mitigated-roberta-snli` model.
16+
- Added support for Flickr30k image retrieval, including a dataset reader, a model, and a training config.
1617

1718
### Fixed
1819

‎allennlp_models/vision/dataset_readers/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,3 +4,4 @@
44
from allennlp_models.vision.dataset_readers.vgqa import VGQAReader
55
from allennlp_models.vision.dataset_readers.vqav2 import VQAv2Reader
66
from allennlp_models.vision.dataset_readers.visual_entailment import VisualEntailmentReader
7+
from allennlp_models.vision.dataset_readers.flickr30k import Flickr30kReader

‎allennlp_models/vision/dataset_readers/flickr30k.py‎

Lines changed: 480 additions & 0 deletions
Large diffs are not rendered by default.

‎allennlp_models/vision/dataset_readers/vision_reader.py‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -96,11 +96,13 @@ def __init__(
9696
max_instances: Optional[int] = None,
9797
image_processing_batch_size: int = 8,
9898
write_to_cache: bool = True,
99+
manual_distributed_sharding: bool = True,
100+
manual_multiprocess_sharding: bool = True,
99101
) -> None:
100102
super().__init__(
101103
max_instances=max_instances,
102-
manual_distributed_sharding=True,
103-
manual_multiprocess_sharding=True,
104+
manual_distributed_sharding=manual_distributed_sharding,
105+
manual_multiprocess_sharding=manual_multiprocess_sharding,
104106
)
105107

106108
# tokenizers and indexers

‎allennlp_models/vision/models/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
from allennlp_models.vision.models.nlvr2 import Nlvr2Model
22
from allennlp_models.vision.models.vision_text_model import VisionTextModel
33
from allennlp_models.vision.models.visual_entailment import VisualEntailmentModel
4+
from allennlp_models.vision.models.vilbert_image_retrieval import ImageRetrievalVilbert
45
from allennlp_models.vision.models.vilbert_vqa import VqaVilbert
56
from allennlp_models.vision.models.heads.vqa_head import VqaHead
67
from allennlp_models.vision.models.heads.visual_entailment_head import VisualEntailmentHead

‎allennlp_models/vision/models/vilbert_image_retrieval.py‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
import logging
2+
from typing import Dict
3+
4+
from overrides import overrides
5+
import torch
6+
7+
from allennlp.data import TextFieldTensors, Vocabulary
8+
from allennlp.models.model import Model
9+
from allennlp.modules.transformer import (
10+
TransformerEmbeddings,
11+
ImageFeatureEmbeddings,
12+
BiModalEncoder,
13+
)
14+
from allennlp.training.metrics import CategoricalAccuracy
15+
from torch.nn import CrossEntropyLoss
16+
17+
from allennlp_models.vision.models.vision_text_model import VisionTextModel
18+
19+
logger = logging.getLogger(__name__)
20+
21+
22+
@Model.register("vilbert_ir")
23+
@Model.register("vilbert_ir_from_huggingface", constructor="from_huggingface_model_name")
24+
class ImageRetrievalVilbert(VisionTextModel):
25+
"""
26+
Model for image retrieval task based on the VilBERT paper.
27+
28+
# Parameters
29+
30+
vocab : `Vocabulary`
31+
text_embeddings : `TransformerEmbeddings`
32+
image_embeddings : `ImageFeatureEmbeddings`
33+
encoder : `BiModalEncoder`
34+
pooled_output_dim : `int`
35+
fusion_method : `str`, optional (default = `"mul"`)
36+
dropout : `float`, optional (default = `0.1`)
37+
label_namespace : `str`, optional (default = `answers`)
38+
k: `int`, optional (default = `1`)
39+
"""
40+
41+
def __init__(
42+
self,
43+
vocab: Vocabulary,
44+
text_embeddings: TransformerEmbeddings,
45+
image_embeddings: ImageFeatureEmbeddings,
46+
encoder: BiModalEncoder,
47+
pooled_output_dim: int,
48+
fusion_method: str = "mul",
49+
dropout: float = 0.1,
50+
k: int = 1,
51+
*,
52+
ignore_text: bool = False,
53+
ignore_image: bool = False,
54+
) -> None:
55+
super().__init__(
56+
vocab,
57+
text_embeddings,
58+
image_embeddings,
59+
encoder,
60+
pooled_output_dim,
61+
fusion_method,
62+
dropout,
63+
is_multilabel=False,
64+
ignore_text=ignore_text,
65+
ignore_image=ignore_image,
66+
)
67+
self.classifier = torch.nn.Linear(pooled_output_dim, 1)
68+
69+
self.top_1_acc = CategoricalAccuracy()
70+
self.top_5_acc = CategoricalAccuracy(top_k=5)
71+
self.top_10_acc = CategoricalAccuracy(top_k=10)
72+
self.loss = CrossEntropyLoss()
73+
74+
self.k = k
75+
76+
@overrides
77+
def forward(
78+
self, # type: ignore
79+
box_features: torch.Tensor,
80+
box_coordinates: torch.Tensor,
81+
box_mask: torch.Tensor,
82+
caption: TextFieldTensors,
83+
label: torch.Tensor,
84+
) -> Dict[str, torch.Tensor]:
85+
batch_size = box_features.shape[0]
86+
87+
if self.training:
88+
# Shape: (batch_size, num_images, pooled_output_dim)
89+
pooled_output = self.backbone(box_features, box_coordinates, box_mask, caption)[
90+
"pooled_boxes_and_text"
91+
]
92+
93+
# Shape: (batch_size, num_images)
94+
logits = self.classifier(pooled_output).squeeze(-1)
95+
probs = torch.softmax(logits, dim=-1)
96+
else:
97+
with torch.no_grad():
98+
# Shape: (batch_size, num_images, pooled_output_dim)
99+
pooled_output = self.backbone(box_features, box_coordinates, box_mask, caption)[
100+
"pooled_boxes_and_text"
101+
]
102+
103+
# Shape: (batch_size, num_images)
104+
logits = self.classifier(pooled_output).squeeze(-1)
105+
probs = torch.softmax(logits, dim=-1)
106+
107+
outputs = {"logits": logits, "probs": probs}
108+
outputs = self._compute_loss_and_metrics(batch_size, outputs, label)
109+
return outputs
110+
111+
@overrides
112+
def _compute_loss_and_metrics(
113+
self,
114+
batch_size: int,
115+
outputs: torch.Tensor,
116+
labels: torch.Tensor,
117+
):
118+
outputs["loss"] = self.loss(outputs["logits"], labels) / batch_size
119+
self.top_1_acc(outputs["logits"], labels)
120+
self.top_5_acc(outputs["logits"], labels)
121+
self.top_10_acc(outputs["logits"], labels)
122+
return outputs
123+
124+
@overrides
125+
def get_metrics(self, reset: bool = False) -> Dict[str, float]:
126+
return {
127+
"top_1_acc": self.top_1_acc.get_metric(reset),
128+
"top_5_acc": self.top_5_acc.get_metric(reset),
129+
"top_10_acc": self.top_10_acc.get_metric(reset),
130+
}
131+
132+
@overrides
133+
def make_output_human_readable(
134+
self, output_dict: Dict[str, torch.Tensor]
135+
) -> Dict[str, torch.Tensor]:
136+
return output_dict
137+
138+
default_predictor = "vilbert_ir"

‎allennlp_models/vision/predictors/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
1+
from allennlp_models.vision.predictors.vilbert_ir import VilbertImageRetrievalPredictor
12
from allennlp_models.vision.predictors.vilbert_vqa import VilbertVqaPredictor
23
from allennlp_models.vision.predictors.visual_entailment import VisualEntailmentPredictor

‎allennlp_models/vision/predictors/vilbert_ir.py‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
from typing import List, Dict
2+
3+
from overrides import overrides
4+
import numpy
5+
6+
from allennlp.common.file_utils import cached_path
7+
from allennlp.common.util import JsonDict
8+
from allennlp.data import Instance
9+
from allennlp.data.fields import LabelField
10+
from allennlp.predictors.predictor import Predictor
11+
12+
13+
@Predictor.register("vilbert_ir")
14+
class VilbertImageRetrievalPredictor(Predictor):
15+
def predict(self, image: str, caption: str) -> JsonDict:
16+
image = cached_path(image)
17+
return self.predict_json({"caption": caption, "image": image})
18+
19+
@overrides
20+
def _json_to_instance(self, json_dict: JsonDict) -> Instance:
21+
from allennlp_models.vision.dataset_readers.flickr30k import Flickr30kReader
22+
23+
caption = json_dict["caption"]
24+
image = cached_path(json_dict["image"])
25+
if isinstance(self._dataset_reader, Flickr30kReader):
26+
return self._dataset_reader.text_to_instance(caption, image, use_cache=False)
27+
else:
28+
raise ValueError(
29+
f"Dataset reader is of type f{self._dataset_reader.__class__.__name__}. "
30+
f"Expected {Flickr30kReader.__name__}."
31+
)
32+
33+
@overrides
34+
def predictions_to_labeled_instances(
35+
self, instance: Instance, outputs: Dict[str, numpy.ndarray]
36+
) -> List[Instance]:
37+
new_instance = instance.duplicate()
38+
label = numpy.argmax(outputs["probs"])
39+
new_instance.add_field("label", LabelField(int(label), skip_indexing=True))
40+
return [new_instance]

‎test_fixtures/vision/flickr30k/experiment.jsonnet‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
local model_name = "epwalsh/bert-xsmall-dummy";
2+
3+
{
4+
"dataset_reader": {
5+
"type": "flickr30k",
6+
"image_dir": "test_fixtures/vision/images/flickr30k",
7+
"data_dir": "test_fixtures/vision/flickr30k/sentences",
8+
"image_loader": "torch",
9+
"image_featurizer": "null",
10+
"featurize_captions": false,
11+
"region_detector": {
12+
"type": "random",
13+
"seed": 322
14+
},
15+
"tokenizer": {
16+
"type": "pretrained_transformer",
17+
"model_name": model_name
18+
},
19+
"token_indexers": {
20+
"tokens": {
21+
"type": "pretrained_transformer",
22+
"model_name": model_name
23+
}
24+
}
25+
},
26+
"train_data_path": "test_fixtures/vision/flickr30k/tiny-dev.txt",
27+
"validation_data_path": "test_fixtures/vision/flickr30k/tiny-dev.txt",
28+
"model": {
29+
"type": "vilbert_ir",
30+
"text_embeddings": {
31+
"vocab_size": 250,
32+
"embedding_size": 20,
33+
"pad_token_id": 0,
34+
"max_position_embeddings": 512,
35+
"type_vocab_size": 2,
36+
"dropout": 0.0
37+
},
38+
"image_embeddings": {
39+
"feature_size": 10,
40+
"embedding_size": 200
41+
},
42+
"encoder": {
43+
# text
44+
"hidden_size1": 20,
45+
"num_hidden_layers1": 1,
46+
"intermediate_size1": 40,
47+
"num_attention_heads1": 1,
48+
"attention_dropout1": 0.1,
49+
"hidden_dropout1": 0.1,
50+
"biattention_id1": [0, 1],
51+
"fixed_layer1": 0,
52+
53+
# vision
54+
"hidden_size2": 200,
55+
"num_hidden_layers2": 1,
56+
"intermediate_size2": 50,
57+
"num_attention_heads2": 1,
58+
"attention_dropout2": 0.0,
59+
"hidden_dropout2": 0.0,
60+
"biattention_id2": [0, 1],
61+
"fixed_layer2": 0,
62+
63+
"combined_num_attention_heads": 2,
64+
"combined_hidden_size": 200,
65+
"activation": "gelu",
66+
},
67+
"pooled_output_dim": 100,
68+
"fusion_method": "sum",
69+
},
70+
"data_loader": {
71+
"batch_size": 4
72+
},
73+
"trainer": {
74+
"optimizer": {
75+
"type": "huggingface_adamw",
76+
"lr": 0.00005
77+
},
78+
"num_epochs": 1,
79+
}
80+
}

‎test_fixtures/vision/flickr30k/experiment_from_huggingface.jsonnet‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
local model_name = "epwalsh/bert-xsmall-dummy";
2+
{
3+
"dataset_reader": {
4+
"type": "flickr30k",
5+
"image_dir": "test_fixtures/vision/images/flickr30k",
6+
"data_dir": "test_fixtures/vision/flickr30k/sentences",
7+
"image_loader": "torch",
8+
"image_featurizer": "null",
9+
"featurize_captions": false,
10+
"region_detector": {
11+
"type": "random",
12+
"seed": 322
13+
},
14+
"tokenizer": {
15+
"type": "pretrained_transformer",
16+
"model_name": model_name
17+
},
18+
"token_indexers": {
19+
"tokens": {
20+
"type": "pretrained_transformer",
21+
"model_name": model_name
22+
}
23+
}
24+
},
25+
"train_data_path": "test_fixtures/vision/flickr30k/tiny-dev.txt",
26+
"validation_data_path": "test_fixtures/vision/flickr30k/tiny-dev.txt",
27+
"model": {
28+
"type": "vilbert_ir_from_huggingface",
29+
"model_name": model_name,
30+
"image_feature_dim": 10,
31+
"image_num_hidden_layers": 1,
32+
"image_hidden_size": 200,
33+
"image_num_attention_heads": 1,
34+
"image_intermediate_size": 50,
35+
"image_attention_dropout": 0.0,
36+
"image_hidden_dropout": 0.0,
37+
"image_biattention_id": [0, 1],
38+
"image_fixed_layer": 0,
39+
40+
"text_biattention_id": [0, 1],
41+
"text_fixed_layer": 0,
42+
43+
"combined_hidden_size": 200,
44+
"combined_num_attention_heads": 4,
45+
46+
"pooled_output_dim": 100,
47+
"fusion_method": "sum",
48+
"pooled_dropout": 0.0,
49+
},
50+
"data_loader": {
51+
"batch_size": 32
52+
},
53+
"trainer": {
54+
"optimizer": {
55+
"type": "huggingface_adamw",
56+
"lr": 0.00005
57+
},
58+
"num_epochs": 1,
59+
}
60+
}

0 commit comments

Comments
 (0)