Integrated Vision & Language

Video Description using Bidirectional Recurrent Neural Networks

 

Members: Marc Bolaños, Petia Radeva, Álvaro Perís (UPV), Francisco Casacuberta (UPV)

screen-shot-2016-10-28-at-17-43-00Although traditionally used in the machine translation field, the encoder-decoder framework has been recently applied for the generation of video and image descriptions. The combination of Convolutional and Recurrent Neural Networks in these models has proven to outperform the previous state of the art, obtaining more accurate video descriptions. In this work, we propose pushing further this model by introducing two contributions into the encoding stage. First, producing richer image representations by combining object and location information from Convolutional Neural Networks and second, introducing Bidirectional Recurrent Neural Networks for capturing both forward and backward temporal relationships in the input frames.

References:

Egocentric Image Retrieval with Convolutional Neural Networks

Members: Gabriel de Oliveira, Mariella Dimiccoli, Petia Radeva

screen-shot-2016-10-28-at-18-05-44

Recent advances in lifelogging technologies, and in particular, in the field of wearable cameras, have made possible to capture continuously our daily life from a first-person point of view and in a free-hand fashion. However, given the large amount of images captured and the rate to which they  increase (up 2000 images per day), there is a strong need for efficient and scalable indexing and retrieval systems over egocentric images. To cope with those requirements, we develop a full Content-Based Image Retrieval system based on Convolutional Neural Network (CNN) features. In our approach, we use egocentric images to create a Lucene index with off-the-shelf features extracted from a pre-trained CNN. The extracted features are integrated into Solr, an open-source, state-of-the-art inverted index search platform. Finally, we provide a web-based prototype for egocentric image search and retrieval and tested its performances on the EDUB egocentric dataset.

References:

Contact:

petia-dot-ivanova-at-ub-dot-edu
Facultat de Matemàtiques i Informàtica, Universitat de Barcelona, Av. Gran Via de les Corts Catalanes 585, 08007 Barcelona, Spain
Web Design MymensinghPremium WordPress ThemesWeb Development

Postdoc on Deep learning and Computer Vision

hiring2A one-year postdoc position is open withnin an European project (https://euraxess.ec.europa.eu/jobs/246415)

Rememory on the press: Una fotografia cada trenta segons per ajudar a fixar els records

Rememory

Rememory on the press:  Una fotografia cada trenta segons per ajudar a fixar els records

El José Maria Carrillo porta una petita càmera penjada al coll que cada trenta segons fa una fotografia. Del cafè que pren amb el seu amic. De la passejada diària que fa. De la visita a l’hospital. De l’excursió amb la seva dona i el seu fill. Si alguna cosa no la vol fotografiar, simplement tapa la càmera amb la mà. Però el José María està satisfet amb l’experiència. És un dels pacients que formen part d’un assaig clínic d’un projecte innovador del Consorci Sanitari de Terrassa -amb la col·laboració de la UB i la Fundació Avan, i finançat per La Marató de TV3 - per millorar la capacitat funcional de les persones amb problemes de memòria. S’ha presentat en el marc del congrés Women 360º.

ReMemory és un programa d’entrenament cognitiu pioner per a persones amb deteriorament cognitiu lleu basat en el r egistre de la vida diària mitjançant fotografies que després serviran per estimular la memòria d’aquests pacients, que estan en la fase prèvia a la malaltia d’ Alzheimer. Són persones encara funcionalment actives. És a dir, encara poden realitzar per si sols activitats de la vida quotidiana. En les fases inicials d’aquesta demència neurodegenerativa, que a Catalunya afecta unes 86.000 persones, el pacient té pèrdues d’atenció i de memòria. “El que fem és reactivar les xarxes neuronals perquè la seva capacitat funcional duri més temps. No modifiquem el procés de la malaltia sinó que allarguem el temps durant el qual poden ser independents”, explica Maite Garolera, investigadora en neuropsicologia del Consorci Sanitari de Terrassa.

International Workshop on Social Signal Processing and Beyond, ICIAP’2017

descarregaInternational Workshop on Social Signal Processing and Beyond ********************************************************************** http://www.ub.edu/cvub/SSPandBE/index.html September 11, 2017, Catania, Italy in association with ICIAP 2017 (http://www.iciap2017.com/)

CFP: LTA2017 – Second International Workshop on Lifelogging Tools and Applications – a workshop at ACM MM 2017

lifelogging-218x150Given the increasing quantities of personal data being gathered by individuals, the concept of a digital library of rich multimedia and sensory content for every individual is becoming a reality and fast becoming a mainstream topic for multimedia research. This is often referred to as lifelogging and there are significant challenges to be addressed in the area, concerning the gathering, enriching, searching and accessing of lifelog data.

Seminary of Adriana Romero and Michal Drozdal: “Towards AI personalized medicine”

tiramisuIn recent years, deep learning has achieved remarkable results in fields such as: computer vision, speech recognition and natural language processing. This DL revolution is slowly reaching the challenging problems of the medical domain, opening the doors for personalized medicine. Medical domain is characterized by high variability of data including text, imaging, and genomic data. In this talk, we will present recent advances in two domains of medical data: imaging and genomics. First, we will introduce a simple, yet powerful pipeline for medical image segmentation that combines Fully Convolutional Networks (FCNs) with Fully Convolutional Residual Networks (FC-ResNets). We propose and examine a design that takes particular advantage of recent advances in the understanding of both Convolutional Neural Networks as well as ResNets. Our approach focuses upon the importance of a trainable pre-processing when using FC-ResNets and we show that a low-capacity FCN model can serve as a pre-processor to normalize medical input data. We show that using this pipeline, we exhibit state-of-the-art performance on the challenging Electron Microscopy benchmark, when compared to other 2D methods. We improve segmentation results on CT images of liver lesions, when contrasting with standard FCN methods. Moreover, when applying our 2D pipeline on a challenging 3D MRI prostate segmentation challenge we reach results that are competitive even when compared to 3D methods. The obtained results illustrate the strong potential and versatility of the pipeline by achieving highly accurate results on multi-modality images from different anatomical regions and organs. Second, we will introduce a novel deep learning architecture to address the challenges posed by genomic data, where the number of input features can be orders of magnitude larger than the number of training examples, making it difficult to avoid overfitting, even when using the known regularization techniques. Improving the ability of deep learning to handle such datasets could have an important impact in medical research, more specifically in precision medicine, where high-dimensional data regarding a particular patient is used to make predictions of interest. We propose a novel neural network parameterization, that we call Diet Networks, which considerably reduces the number of free parameters in the model. The Diet Networks parametrization is based on the idea that we can first learn or provide an embedding for each input feature and then learn how to map a feature's representation to the parameters linking the value of the feature to each of the hidden units of the classifier network. We experiment on a population stratification task of interest to medical studies and show that the proposed approach can significantly reduce both the number of parameters and the error rate of the classifier. This work was accepted at ICLR 2017.

Open postdoc position

we-are-hiringOpen Postdoctoral fellowship/s in the Dept. de Matemàtiques i Informàtica at Universitat de Barcelona. The Dept. de Matemàtiques i Informàtica (Mathematics and Computer Science) is looking for and willing to support excellent postdoctoral researchers in the fields of Machine Learning, Computer Vision and Human-Computer Interaction who are interested in applying for a Beatriu de Pinós (BP) 2016 fellowship so as to conduct a two-year postdoc at Universitat de Barcelona. The purpose of the Beatriu de Pinós programme is to award 60 individuals grants for the hiring and incorporation of postdoctoral research staff into the Catalan science and technology system. These grants are designed for the incorporation of young researchers (who obtained their PhD between 2007 and 2014 and have not resided or worked in Spain for more than 12 months in the three years prior to date of submission of the application), so that they can improve their professional prospects and obtain an independent research position. Candidates must carry out a research and training project for the entire period of the grant, one that will allow them to progress in the development of their professional careers. Please check the website of the BP programme* for further information about this fellowship. Some of the specific projects, we are working, include: + Machine Learning: Deep Learning for time series analysis, Supervised Online Learning Algorithms, Bayesian statistics and deep learning. + Computer Vision: Visual Lifelogging and Egocentric Vision, Neuroimage processing, Computer Vision for Food Analysis, Deep learning and Image Aesthetics, Ultrasound image analysis. + Human-Computer Interaction: ageing / older people, interfaces for people with mild dementia or with aphasia, universal design of STEM documents Deadline: 0112/2016. For further information about this postdoctoral opportunity please feel free to contact us: Petia Radeva (petia.ivanova@ub.edu or radevap@gmail.com), www.ub.edu/cvub, www.cvc.uab.es/people/petia

Petia Radeva – invited speaker to the workshop “Humanitarian and social science: from the university to the enterprise”

jornadaPetia Radeva has been invited speaker to the workshop “Humanitarian and social science: from the university to the enterprise” 17 of November, 2016, Faculty of Filology, University of Barcelona.

Best paper award at CIAPR’2016 to our paper: “Deep Learning Features for Wireless Capsule Endoscopy Analysis”, by Santi Segui, Michal Drozdzal, Guillem Pascual, Carolina Malagelada, Fernando Azpiroz, Petia Radeva and Jordi Vitrià

ciarpBest paper award at CIAPR'2016 to our paper: "Deep Learning Features for Wireless Capsule Endoscopy Analysis", by Santi Segui, Michal Drozdzal, Guillem Pascual, Carolina Malagelada, Fernando Azpiroz, Petia Radeva and Jordi Vitrià, Lima Perú, 2016.

Petia Radeva received the International CIARP Award “Aurora Pons Porrata”

aurora-porrataPetia Radeva received the International CIARP Award "Aurora Pons Porrata" in recognition of an outstanding technical contribution to the field of pattern recognition, data mining and related areas, 2016.

Mention prize (II place) for our Application on Automatic Food Recognition in the DKV competence Health4Good!

logodkvWe got the Mention prize (II place) in the DKV competence with our App for Automatic Food Recognition for Healthy Habits Promotion! Congratulations, team!!!

The journal Medical Physics chose figures of our work on Stent analysis to use as a cover on their journal. http://www.medphys.org

medical-physicsThe journal Medical Physics chose figures of our work on Stent analysis to use as a cover on their journal. http://www.medphys.org

Petia Radeva gave the plenary talk at CCIA’2016

petiaPetia Radeva gave a plenary talk "Can Deep Learning and Egocentric Vision for Visual Lifelogging help us eat better?" at the CCIA'2016, organized by Xavier Binefa, UPF, Barcelona, Spain.

GRADIANT award to Beatriz Remeseiro for the best PhD thesis applied to the ICT sector 2016.

bea2The prestigous GRADIANT award was assifned to Beatriz Remeseiro for the best PhD thesis applied to the ICT sector 2016.

3 abstracts accepted at the NIPS Workshop WiML’16.

screen-shot-2016-10-28-at-16-48-543 abstracts accepted at the NIPS Workshop WiML'16. Congratulations, Mariella, Maya and Beatriz!

Dr. Giovanni Maria Farinella is visiting us on 17 of November, 2016

screen-shot-2016-10-28-at-16-36-23Dr. Giovanni Maria Farinella from the Department of Mathematics and Computer Science - University of Catania is visiting us on 17 of November, 2016. He will give us a lecture on their egocentric projects.

Offers of 4 grants for Master Students

José M. Álvarez’s seminary

screen-shot-2016-10-28-at-16-31-36On 8 of February, 2016 José M. Álvarez is visiting CVUB giving a seminaty with title: Compacting ConvNets for end to end Learning.

Michal Drozdal received the award “Pioner 2015” for his PhD thesis “Sequential image analysis for computer-aided wireless endoscopi”, by the Institució CERCA.

Premiats2015Michal Drozdzal is one of the 5 researchers who received the Pioneer Award for her doctoral thesis "Sequential image analysis for computer-aided endoscopi wireless". This is the second edition of the competition promoted by CERCA. This year a total of nineteen researchers (10 males and 9 females) participated from thirteen centres nearby. By areas, there have been 3 science projects; 7 of medical sciences and health; 3 of 6 engineering and life sciences. The jury appreciated the advanced technology that solves problems linked to the endoscopy with a significant population component and a business link which enables a good transfer of knowledge generated.

Re-Memory presented at the CCCB exposition “Human+”

Rememory-fitxaRe-memory: Cognitive Training based on autobiograohic records to exercise memory Exposition “+HUMANS”, 17 December 2015, CCB, Barcelona Spain The majority of patients with mild cognitive impairment develop dementia and Alzheimer's disease. Re-memory, a project financed by the Foudation “La Marató de TV3”, study a new entrenament cognitiu based on the concept of lifelogging to exrcise memory. Re-memory works in the following way: the patient carries a camera that capture automatically images from all locations visited, events in which the wearer participates, the activities that he/she participated and persons he/she interacted with. Inspired in the use of wearable cameras for first cases of amnesia, els membres of the Re-memory project create a cogntive program for training based on the autobiographic reexperimenting to positively impact the cognition and improve the memory and function of peoplewith mild cognitive impairment.

At NVIDIA’s #GTC15, our endoluminal image analysis work was presented during the keynote.

nvidiaOur work on using Deep Learning techniques for endoluminal image analysis was presented as example at NVIDIA’s GPU Technical Conference (http://www.gputechconf.com/).