Research

2023

1. ICCV

   Geometric Contrastive Learning

   Yeskendir Koishekenov, Sharvaree Vadgama*, Ricardo Valperga*, and 1 more author

   In ICCV 2023 - Visual Inductive Priors for Data-Efficient Deep Learning workshop (Oral), Oct 2023

   Abs Website

   Contrastive learning has been a long-standing research area due to its versatility and importance in learning representations. Recent works have shown improved results if the learned representations are constrained to be on a hypersphere. However, this \textitprior geometric constraint is not fully utilized during training. In this work, we propose making use of geodesic distances on the hypersphere to learn contrasts between representations. Through empirical results, we show that this contrastive learning approach improves downstream tasks across different contrastive learning frameworks. We show that having geometric inductive priors perform even better in contrastive learning if used along with other correct geometric information.

2. ACL

   Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model

   Yeskendir Koishekenov, Alexandre Berard, and Vassilina Nikoulina

   In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Jul 2023

   Abs arXiv Code Website

   The recently released NLLB-200 is a set of multilingual Neural Machine Translation models that cover 202 languages. The largest model is based on a Mixture of Experts architecture and achieves SoTA results across many language pairs. It contains 54.5B parameters and requires at least four 32GB GPUs just for inference.In this work, we propose a pruning method that enables the removal of up to 80% of experts without further finetuning and with a negligible loss in translation quality, which makes it feasible to run the model on a single 32GB GPU. Further analysis suggests that our pruning metrics can identify language-specific experts.

3. ICLR

   An Exploration of Conditioning Methods in Graph Neural Networks

   Yeskendir Koishekenov, and Erik J Bekkers

   In ICLR 2023 - Machine Learning for Drug Discovery workshop, May 2023

   Abs Code Website

   The flexibility and effectiveness of message passing based graph neural networks (GNNs) induced considerable advances in deep learning on graph-structured data. In such approaches, GNNs recursively update node representations based on their neighbors and they gain expressivity through the use of node and edge attribute vectors. E.g., In computational tasks such as physics and chemistry usage of edge attributes such as relative position or distance proved to be essential. In this work, we address not what kind of attributes to use, but how to condition on this information to improve model performance. We consider three types of conditioning; weak, strong, and pure, which respectively relate to concatenation-based conditioning, gating, and transformations that are causally dependent on the attributes. This categorization provides a unifying viewpoint on different classes of GNNs, from separable convolutions to various forms of message passing networks. We provide an empirical study on the effect of conditioning methods in several tasks in computational chemistry.

4. AAAI

   Reducing Over-smoothing in Graph Neural Networks Using Relational Embeddings

   Yeskendir Koishekenov

   Accepted to The Ninth International Workshop on Deep Learning on Graphs: Method and Applications (DLG-AAAI’23), Feb 2023

   Abs arXiv Code Website

   Graph Neural Networks (GNNs) have achieved a lot of success with graph-structured data. However, it is observed that the performance of GNNs does not improve (or even worsen) as the number of layers increases. This effect has known as over-smoothing, which means that the representations of the graph nodes of different classes would become indistinguishable when stacking multiple layers. In this work, we propose a new simple, and efficient method to alleviate the effect of the over-smoothing problem in GNNs by explicitly using relations between node embeddings. Experiments on real-world datasets demonstrate that utilizing node embedding relations makes GNN models such as Graph Attention Network more robust to over-smoothing and achieves better performance with deeper GNNs. Our method can be used in combination with other methods to give the best performance. GNN applications are endless and depend on the user’s objective and the type of data that they possess. Solving over-smoothing issues can potentially improve the performance of models on all these tasks.

2022

1. ReScience

   Reproducibility study of" Data-Driven Methods for Balancing Fairness and Efficiency in Ride-Pooling"

   Sarah De Boer*, Radu Alexandru Cosma*, Lukas Knobel*, and 2 more authors

   In ML Reproducibility Challenge 2021 (Fall Edition), Feb 2022

   Abs PDF Code

   Our work attempts to verify two methods to mitigate forms of inequality in ride‐pooling platforms proposed in the paper Data-Driven Methods for Balancing Fairness and Efficiency in Ride-Pooling: (1) integrating fairness constraints into the objective functions and (2) redistributing income of drivers. We extend this paper by testing for robustness to a change in the neighbourhood selection process by using actual Manhattan neighbour‐ hoods and we use corresponding demographic data to examine differences in service based on ethnicity.

2020

1. ECCV

   Aim 2020 challenge on learned image signal processing pipeline

   Andrey Ignatov, Radu Timofte, Zhilu Zhang, and 8 more authors

   In Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16, Feb 2020

   Abs arXiv

   This paper reviews the second AIM learned ISP challenge and provides the description of the proposed solutions and results. The participating teams were solving a real-world RAW-to-RGB mapping problem, where to goal was to map the original low-quality RAW images captured by the Huawei P20 device to the same photos obtained with the Canon 5D DSLR camera. The considered task embraced a number of complex computer vision subtasks, such as image demosaicing, denoising, white balancing, color and contrast correction, demoireing, etc. The target metric used in this challenge combined fidelity scores (PSNR and SSIM) with solutions’ perceptual results measured in a user study. The proposed solutions significantly improved the baseline results, defining the state-of-the-art for practical image signal processing pipeline modeling.