Faster testing code

[AshwinParanjape]

• Creating the softmax_W tensor on CPU makes it transfer the tensor from CPU to GPU for every batch during test time (where full softmax is used), slowing down the code by 5x or so.
• Transposing softmax_W is a large operation compared to transposing input and output vectors, increasing the speed of full softmax further

By profiling using timeline from Tensorflow we observe the following
Before -

After -

The big MEMCPYHtoD and big Transpose before MatMul are gone

[matt-peters]

Thanks for the PR!

I put the softmax variables on CPU for training as they occupy a lot of memory and were overwhelming the poor 8 GB GeForce GTX 1080 GPUs I originally used to train the ELMo model.

Can you make the device placement for the softmax configurable? Or place them on GPU for testing only (by checking the value of self.is_training)?