Importance Sampling with exponential running average thresholding

Working with global statistics instead of per-batch statistics

In my earlier article (part 1), I discussed implementation of importance sampling, based on per-batch statistics. There, a sample with loss value in the top nth-percentile of its corresponding batch was filtered for training.

Now, the shortcoming of the above approach is that it is possible that most batches contains only simple samples. Even if we filter the batch, the filtered samples are still simple enough for the model. Therefore, a filtering scheme contingent on individual batch statistics is unable to fully exploit the benefits of importance sampling.

With that thought, I wanted to employ whole dataset dependent statistics to filter training samples. This required keeping track of statistics over multiple batches. Unaware of a tensorflow/keras recommended way for this, I have implemented a custom tf.keras.layers.Layer (code below) to track exponential moving average (EMA) of top n-th percentile loss values from all the batches.

With the custom layer defined, loss thresholding can be included as follows.

Therefore, going forward I plan to implement loss thresholding based on running statistics on last few training batches (part 3). Moreover, think of ways to take care of other shortcomings mentioned in the naïve approach.

Please share your feedback and suggestions for further improving on this.

If you find stories like these valuable and would like to support me as a writer, please consider following me or signing up for Medium membership.

Related posts:

AdultSearch is an app designed to get you laid. We’re special when you compare us to your standard escort apps or hookup sites. On AdultSearch, there are just regular women who want to hookup and are…

Hindi and English Poem. “That Eid Moon” is published by Priyanka Srivastava in Polyglot Poetry.