Wednesday, June 21, 2017

Videos: Structured Regularization for High-Dimensional Data Analysis: Compressed Sensing: Structure and Imaging & Matrix and graph estimation




Lectures 1: Compressed Sensing: Structure and Imaging
   
 Lectures 2: Compressed Sensing: Structure and Imaging Anders Hansen (Cambridge) 

Lectures 1 and 2: Compressed Sensing: Structure and Imaging Abstract: The above heading is the title of a new book to be published by Cambridge University Press. In these lectures I will cover some of the main issues discussed in this monograph/textbook. In particular, we will discuss how the key to the success of compressed sensing applied in imaging lies in the structure. For example images are not just sparse in an X-let expansion, they have a very specific sparsity structure in levels according to the X-let scales. Similarly, when considering Total Variation, the gradient coefficients are also highly structured. Moreover, in most realistic sampling scenarios, the sampling operator combined with any X-let transform yields a matrix with a very specific coherence structure. The key to successfully use compressed sensing is therefore to understand how to utilise these structures in an optimal way, in particular in the sampling procedure. In addition, as the coherence and sparsity structures have very particular asymptotic behaviour, the performance of compressed sensing varies greatly with dimension, and so does the optimal way of sampling. Fortunately, there is now a developed theory that can guide the user in detail on how to optimise the use of compressed sensing in inverse and imaging problems. I will cover several of the key aspects of the theory accompanied with real-world examples from Magnetic Resonance Imaging (MRI), Nuclear Magnetic Resonance (NMR), Surface Scattering, Electron Microscopy, Fluorescence Microscopy etc. Recommended readings: (lectures 1 and 2) Chapter 4, 6 and 12 in “A mathematical introduction to compressed sensing” (Foucard/Rauhut) Breaking the coherence barrier: A new theory for compressed sensing On asymptotic structure in compressed sensing Structure dependent sampling in compressed sensing: theoretical guarantees for tight frames
   

Andrea Montanari (Stanford): Matrix and graph estimation 

 Abstract: Many statistics and unsupervised learning problems can be formalized as estimating a structured matrix or a graph from noisy or incomplete observations. These problems present a large variety of challenges, and an intriguing interplay between computational and statistical barriers. I will provide an introduction to recent work in the area, with an emphasis on general methods and unifying themes. 1) Random matrix theory and spectral methods. 2) The semidefinite programming approach to graph clustering. 3) Local algorithms and graphical models. The hidden clique problem. 4) Non-negative matrix factorization.






Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Tuesday, June 20, 2017

Learning Deep ResNet Blocks Sequentially using Boosting Theory

Here is another way of cutting down Deep Neural Nets training time.

Learning Deep ResNet Blocks Sequentially using Boosting Theory by Furong Huang, Jordan Ash, John Langford, Robert Schapire
Deep neural networks are known to be difficult to train due to the instability of back-propagation. A deep \emph{residual network} (ResNet) with identity loops remedies this by stabilizing gradient computations. We prove a boosting theory for the ResNet architecture. We construct T weak module classifiers, each contains two of the T layers, such that the combined strong learner is a ResNet. Therefore, we introduce an alternative Deep ResNet training algorithm, \emph{BoostResNet}, which is particularly suitable in non-differentiable architectures. Our proposed algorithm merely requires a sequential training of T "shallow ResNets" which are inexpensive. We prove that the training error decays exponentially with the depth T if the \emph{weak module classifiers} that we train perform slightly better than some weak baseline. In other words, we propose a weak learning condition and prove a boosting theory for ResNet under the weak learning condition. Our results apply to general multi-class ResNets. A generalization error bound based on margin theory is proved and suggests ResNet's resistant to overfitting under network with l1 norm bounded weights.  






Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Monday, June 19, 2017

Job; Postdoc Opening in Statistical Learning and Optimization, USC

Jason just sent me the following last week:


Dear Igor, 
I am looking for a postdoc. Could you please post the following on Nuit Blanche? BTW, I greatly enjoy reading your blog.

Postdoc Opening in Statistical Learning and Optimization 
Applications are invited for a postdoc position to work with Jason Lee at the University of Southern California. Prospective applicants should have a PhD in statistics, machine learning, signal processing, or applied mathematics. The successful candidate will have the flexibility to choose a focus within non-convex optimization, deep learning, and statistical inference and learning theory. Applications from candidates with a strong background in Optimization, Statistics, and Machine Learning are particularly welcome. 
Applicants are requested to send a CV, 3 representative papers, and the contact details of three references. 
Please send applications and informal inquiries to jasonlee(AT)marshall(DOT)usc(DOT)edu.

Best,
Jason


Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Job: Lecturer / Senior Lecturer in Machine Learning and Computational Intelligence, University of Surrey, UK

Mark just me the folloing last week:

Dear Igor,
This job opportunity may interest some readers of Nuit Blanche?
Best wishes,
Mark
---
Lecturer / Senior Lecturer in Machine Learning and Computational Intelligence
Department of Computer Science, University of Surrey
Salary: GBP 39,324 to GBP 57,674 per annum
Closing Date: Wednesday 28 June 2017
The Department of Computer Science at the University of Surrey invites applications for a post of Lecturer/Senior Lecturer (similar to Assistant/Associate Professor) in Machine Learning and Computational Intelligence. We aim to attract outstanding candidates to the Nature Inspired Computing and Engineering (NICE) Group, who will have strong visions for research, a growing international research profile, a passion for teaching, and who value collaborative research and working in a team. This is an exciting opportunity in a department that is growing its reputation for delivering quality interdisciplinary and applied research based on strong foundations.
The post-holder will enhance or complement one or more of the following areas: evolutionary computation, computational intelligence, machine learning, and computational neuroscience, with applications to data-driven optimisation, data analytics and big data, secure machine learning, self-organising and autonomous systems, healthcare and bioinformatics. It is expected that the post-holder will also contribute to high quality teaching at undergraduate and post-graduate level, for example in data science.
Applicants to the post should have a PhD in a relevant subject or equivalent professional experience. An ability to produce high quality outputs is also required. The appointed candidate will be expected to contribute to all aspects of the Department's activities.
We are looking for individuals that can inspire students through their curiosity for leading-edge aspects of technology. In particular, the teaching duties of the role include delivering high quality teaching to all levels of students, supervising undergraduate project students and postgraduate dissertations and contributing to the teaching of computational intelligence, machine learning, data science, as well as other practical areas of Computer Science, such as object-oriented programming and advanced algorithms.
The Department of Computer Science embodies the ethos of "applying theory into practice" across its research and teaching activities. Its research activities are focused into two research groups: Nature Inspired Computing and Engineering (NICE) and Secure Systems. The research on evolutionary optimization and computational intelligence is internationally recognized and has a close collaboration across the University and internationally with academia and industry. The department has excellent links with departments across the University, as well as with industry, such as Honda, The Pirbright Institute, National Physical Laboratory, and Moorfields Eye Hospital.
This is a full-time and permanent position. The post is available from September 2017 but there is flexibility in the start date. The University is located close to the Surrey Research Park in Guildford and within easy commuting distance of London.
For further details and an informal discussion please contact Professor Yaochu Jin, Head of the NICE Group, yaochu.jin@surrey.ac.uk, or Professor Mark Plumbley, Head of Department at m.plumbley@surrey.ac.uk.
The University and the Department specifically are committed to building a culturally diverse organisation and strongly encourages applications from female and minority candidates. The Department shares the Athena SWAN ideals with respect to the equality and diversity agenda.
Presentations and interviews will take place on 13 July 2017. During the interview process there will be an opportunity to meet members of the NICE group in the evening of 12 July 2017 for an informal dinner and to meet with other colleagues within the Department on an informal basis.
For further details and information on how to apply, visit https://jobs.surrey.ac.uk/039217
--
Prof Mark D Plumbley
Interim Head of Department of Computer Science
Professor of Signal Processing
Centre for Vision, Speech and Signal Processing (CVSSP)
University of Surrey - Celebrating 50 years in Guildford
Email (Head of Department matters): cs-hod@list.surrey.ac.uk
Email (Other matters): m.plumbley@surrey.ac.uk
University of Surrey, Guildford, Surrey, GU2 7XH, UK



Sunday, June 18, 2017

Sunday Morning Videos: NIPS 2016 workshop on nonconvex optimizations.



Hossein just mentioned this on his twitter:

Here is the list of talks and videos:
  • Nando de Freitas (Learning to Optimize) [Slides] [Video]
  • Morning Poster Spotlight (papers #1 to #8) [Slides] [Video]
  • Jean Lasserre (Moment-LP and Moment-SOS Approaches in Optimization) [Slides] [Video]
  • Surya Ganguli (Non-convexity in the error landscape and the expressive capacity of deep neural networks) [Slides] [Video
  • Ryan Adams (Leveraging Structure in Bayesian Optimization) [Slides] [Video]
  • Stefanie Jegelka (Submodular Optimization and Nonconvexity) [Slides] [Video]
  • Suvrit Sra (Taming Non-Convexity via Geometry) [Slides] [Video]
  • Francis Bach (Submodular Functions: from Discrete to Continuous Domains) [Slides] [Video]
  • Panel Discussion [Video]
  • Afternoon Poster Spotlight (papers #9 to #16) [Slides] [Video]





Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Friday, June 16, 2017

FreezeOut: Accelerate Training by Progressively Freezing Layers - implementation -

What initially looks like playing with hyperparameters brings new life to a somewhat older approach. From Alex's tweet:







The early layers of a deep neural net have the fewest parameters, but take up the most computation. In this extended abstract, we propose to only train the hidden layers for a set portion of the training run, freezing them out one-by-one and excluding them from the backward pass. We empirically demonstrate that FreezeOut yields savings of up to 20% wall-clock time during training with 3% loss in accuracy for DenseNets on CIFAR.


DenseNet is at: http://github.com/bamos/densenet.pytorch 
while FreezeOut is here: http://github.com/ajbrock/FreezeOut 




Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Thursday, June 15, 2017

Self-Normalizing Neural Networks - implementation -

There is some commotion on the interweb about this new nonlinear activation function that removes the need for tricks like Batch Normalization and seem to beat recent architectures like Resnets ...and there is the long appendix thingy too. Wow !




Deep Learning has revolutionized vision via convolutional neural networks (CNNs) and natural language processing via recurrent neural networks (RNNs). However, success stories of Deep Learning with standard feed-forward neural networks (FNNs) are rare. FNNs that perform well are typically shallow and, therefore cannot exploit many levels of abstract representations. We introduce self-normalizing neural networks (SNNs) to enable high-level abstract representations. While batch normalization requires explicit normalization, neuron activations of SNNs automatically converge towards zero mean and unit variance. The activation function of SNNs are "scaled exponential linear units" (SELUs), which induce self-normalizing properties. Using the Banach fixed-point theorem, we prove that activations close to zero mean and unit variance that are propagated through many network layers will converge towards zero mean and unit variance -- even under the presence of noise and perturbations. This convergence property of SNNs allows to (1) train deep networks with many layers, (2) employ strong regularization, and (3) to make learning highly robust. Furthermore, for activations not close to unit variance, we prove an upper and lower bound on the variance, thus, vanishing and exploding gradients are impossible. We compared SNNs on (a) 121 tasks from the UCI machine learning repository, on (b) drug discovery benchmarks, and on (c) astronomy tasks with standard FNNs and other machine learning methods such as random forests and support vector machines. SNNs significantly outperformed all competing FNN methods at 121 UCI tasks, outperformed all competing methods at the Tox21 dataset, and set a new record at an astronomy data set. The winning SNN architectures are often very deep. Implementations are available at: github.com/bioinf-jku/SNNs.
Some mention on the web:


Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Wednesday, June 14, 2017

CSjob: Postdoc, Optimisation for Matrix Factorisation, Toulouse, France

Cedric came to visit us yesterday at LightOn and he let us know about his search for a postdoc, here is the announcement:

Postdoc opening Optimisation for matrix factorisation
Project FACTORY 
 

New paradigms for latent factor estimation Announcement Applications are invited for a 2-year postdoc position to work with Cedric Fevotte (CNRS senior scientist) on matrix factorisation techniques for data processing. The position is part of project FACTORY (New paradigms for latent factor estimation), funded by the European Research Council under a Consolidator Grant (2016-2021). The successful candidate will be based in Toulouse, France.  
Project description  
The project concerns matrix factorisation and dictionary learning for data analysis at large, with an emphasis on statistical estimation in mean-parametrised exponential models, non-convex optimisation, stochastic algorithms & approximate inference, representation learning, and applications to audio signal processing, remote sensing & data mining.
The European Research Council offers highly competitive funding for scientific excellence. The successful candidate will enjoy an inspiring and resourceful environment, with the possibility of travelling to conferences and visiting other national or international labs.
More information at http://www.irit.fr/~Cedric.Fevotte/factory/  
Host institution and place of work 
The successful candidate will be employed by the Centre National de la Recherche Scientifique (CNRS, the National Center for Scientific Research). CNRS is the largest state-funded research organisation in France, involved in all scientific fields. FACTORY is hosted by the Institut de Recherche en Informatique de Toulouse (IRIT), a joint laboratory of CNRS and Toulouse universities & engineering schools. IRIT is among the largest computer & information sciences labs in France. Toulouse is the fourth largest city in France, the capital of the Midi-Pyr´en´ees region in the South-West of France, and is praised for its high quality of living. The physical location for the project is the ENSEEIHT campus (Signal & Communications group), in a lively neighbourhood of the city center.  
Candidate profile and application  

Prospective applicants should have a PhD in machine learning, signal processing, applied mathematics, statistics, or a related discipline, good programming skills, and good communication skills in English, both written and oral. The successful candidate will have the flexibility to choose a topic within the range of the project, according to his/her experience and preferences. Applications from candidates with a good background in optimisation or stochastic simulation are particularly encouraged. The net monthly salary is 2300e for researchers with less than 2 years of professional experience after the PhD, and starts from 2700e in other cases. The position comes with health insurance & other social benefits. Applicants are requested to send a CV, a brief statement of research interests and the contact details of two referees in a single PDF file. Applications and informal enquiries are to be emailed to cedric(dot)fevotte(at)irit(dot)fr




Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Tuesday, June 13, 2017

Paris Machine Learning meetup "Hors Série" #13: Automatic Machine Learning



So tonight is going to be the 13th Hors Série of Season 4 of the Paris Machine Learning meetup. This will bring about to a record 23 the number of meetups Franck and I  have made happen this season!

Tonight will be a presentation of two toolboxes around the issue of Automated Machine Learning. One toolbox is by Axel de Romblay and the other is by Alexis Bondu.

There is limited seating however we will have a streaming (see below).

Thank to Sewan for hosting us. Here is the video for the streaming:

Program:
  • 0 - Preview of the slides
    • A preview of the slides is now available:
    • - French version (link)
    • - English version (link)
  • 1  - Introduction
    • How to ?
    • What's important ?
  • 2 - Theories and Baseline to automate Machine Learning
    • Overview of the different approaches to automate Machine Learning (bayesian, ...)
  • 3 - Demo & Coding
    • Edge ML &  MLBox
    • Comparison of these tools with the same dataset
  • 3-1 - MLBox (Machine Learning Box)
    • MLBox is a powerful Automated Machine Learning python library. It provides the following functionalities:
      • - Fast reading and distributed data preprocessing / cleaning / formatting
      • - Highly robust feature selection and leak detection
      • - Accurate hyper-parameter optimization in high-dimensional space
      • - State-of-the art predictive models for classification and regression (Deep Learning, Stacking, LightGBM,...)
    • - Prediction with models interpretation
    • To learn more about this tool and how to get it installed, please refer to:
    • https://github.com/AxeldeRomblay/MLBox
  • 3.2 - Edge ML
    • Edge ML is an Automated Machine Learning software which implements the MODL approach, which is unique in two respects :
      • i) it is highly scalable and avoids empirical optimization of the models by grid-search;
      • ii) it provides accurate and very robust models.
    • To participate in the interactive demonstration, you can install Edge ML :
      • 1 - Install the shareware version: link
      • 2 - Create and download your licence file: link
      • 3 - Copy / Paste your licence file to your home directory
    • 4 - Download the Python wrapper (with demo example): link


Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Monday, June 12, 2017

CATERPILLAR: Coarse Grain Reconfigurable Architecture for Accelerating the Training of Deep Neural Networks

Learning to learn with efficiency in mind.



Accelerating the inference of a trained DNN is a well studied subject. In this paper we switch the focus to the training of DNNs. The training phase is compute intensive, demands complicated data communication, and contains multiple levels of data dependencies and parallelism. This paper presents an algorithm/architecture space exploration of efficient accelerators to achieve better network convergence rates and higher energy efficiency for training DNNs. We further demonstrate that an architecture with hierarchical support for collective communication semantics provides flexibility in training various networks performing both stochastic and batched gradient descent based techniques. Our results suggest that smaller networks favor non-batched techniques while performance for larger networks is higher using batched operations. At 45nm technology, CATERPILLAR achieves performance efficiencies of 177 GFLOPS/W at over 80% utilization for SGD training on small networks and 211 GFLOPS/W at over 90% utilization for pipelined SGD/CP training on larger networks using a total area of 103.2 mm2 and 178.9 mm2 respectively.



h/t Iacopo




Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Printfriendly