Signal processing has entered into today's life on a broad range, from mobile phones, UMTS, xDSL, and digital television to scientific research such as psychoacoustic modeling, acoustic measurements, and hearing prosthesis. Such applications often use time-invariant filters by applying the Fourier transform to calculate the complex spectrum. The spectrum is then multiplied by a function, the so-called transfer function. Such an operator can therefore be called a Fourier multiplier. Real life signals are seldom found to be stationary. Quasi-stationarity and fast-time variance characterize the majority of speech signals, transients in music, or environmental sounds, and therefore imply the need for non-stationary system models. Considerable progress can be achieved by reaching beyond traditional Fourier techniques and improving current time-variant filter concepts through application of the basic mathematical concepts of frame multipliers.

Several transforms, such as the Gabor transform (the sampled version of the Short-Time Fourier Transformation), the wavelet transform, and the Bark, Mel, and Gamma tone filter banks are already in use in a large number of signal processing applications. Generalization of these techniques can be obtained via the mathematical frame theory. The advantage of introducing the frame theory consists particularly in the interpretability of filter and analysis coefficients in terms of frequency and time localization, as opposed to techniques based on orthonormal bases.

One possibility to construct time-variant filters exists through the use of Gabor multipliers. For these operators the result of a Gabor transform is multiplied by a given function, called the time-frequency mask or symbol, followed by re-synthesis. These operators are already used implicitly in engineering applications, and have been investigated as Gabor filters in the fields of mathematics and signal processing theory. If alternative transforms are used, the concept of multipliers can be extended appropriately. So, for example, the concept of wavelet multipliers could be investigated for a wavelet transform.

Different kinds of applications call for different frames. Multipliers can be generalized to the abstract level of frames without any further structure. This concept will be further investigated in this project. Its feasibility will be evaluated in acoustic applications using special cases of Gabor and wavelet systems.

The project goal is to study both the mathematical theory of frame multipliers and their application among selected problems in acoustics. The project is divided into the following subprojects:

- General Frame Multiplier Theory
- Analytic and Numeric Properties of Gabor Multipliers
- Analytic and Numeric Properties of Wavelet Multipliers

- Mathematical Modeling of Auditory Time-Frequency Masking Functions
- Improvement of Head-Related Transfer Function Measurements
- Advanced Method of Sound Absorption Measurements

- H.G. Feichtinger et al., NuHAG, Faculty of Mathematics, University of Vienna
- R. Kronland-Martinet et al., Modélisation, Synthèse et Contrôle des Signaux Sonores et Musicaux of the LMA / CNRS Marseille
- B. Torrésani et al., LATP Université de Provence / CNRS Marseille
- J.P. Antoine et al., FYMA Université Catholique de Louvain

- P. Balazs, J.-P. Antoine, A. Gryboś, "Weighted and Controlled Frames: Mutual relationship and first Numerical Properties", accepted for publication in International Journal of Wavelets, Multiresolution and Information Processing (2009), preprint
- P. Balazs, “Matrix Representation of Bounded Linear Operators By Bessel Sequences, Frames and Riesz Sequence“,SampTA'09, 8th International Conference on Sampling and Applications, May 2009, Marseille, France
- A. Rahimi, P. Balazs, "Multipliers for p-Bessel sequences in Banach spaces", submitted (2009)
- D. Stoeva, P. Balazs, "Unconditional convergence and Invertibility of Multipliers", preprint (2009)
- Monika Dörfler and Bruno Torrésani, “Representation of operators in the time-frequency domain and generalized Gabor multipliers”, J. Fourier Anal. Appl., 2009 (in press)
- Yohan Frutiger: "Multiplicateurs de Gabor pour les transformations sonores" (Gabor Multipliers for sound transformations) Master thesis under the supervision of R. Kronland-Martinet, June 2008
- F. Jaillet, P. Balazs, M. Dörfler and N. Engelputzeder, “On the Structure of the Phase around the Zeros of the Short-Time Fourier Transform”, NAG/DAGA 2009, International Conference on Acoustics, March 2009, Rotterdam, Nederland
- F. Jaillet, P. Balazs and M. Dörfler, “Nonstationary Gabor Frames”, SampTA'09, 8th International Conference on Sampling and Applications, May 2009, Marseille, France
- P. Balazs, B. Laback, G. Eckel, W. Deutsch, "Introducing Time-Frequency Sparsity by Removing Perceptually Irrelevant Components Using a Simple Model of Simultaneous Masking", IEEE Transactions on Audio, Speech and Language Processing (2009), in press
- B. Laback, P. Balazs, G. Toupin, T. Necciari, S. Savel, S. Meunier, S. Ystad and R. Kronland-Martinet, "Additivity of auditory masking using Gaussian-shaped tones", Acoustics'08, Paris, 29.06.-04.07.2008 (03.07.2008)
- B. Laback, P. Balazs, T. Necciari, S. Savel, S. Ystad, S. Meunier and R. Kronland-Martinet, "Additivity of auditory masking for Gaussian-shaped tone pulses", preprint
- Anaïk Olivero: "Expérimentation des multiplicateurs temps-échelle" (On the time-scale multipliers) Master thesis under the supervision of R. Kronland-Martinet and B. Torrésani, June 2008

The applications involving signal processing algorithms (like adaptive or time variant filters) are numerous. If the STFT, the Short Time Fourier Transformation, is used in its sampled version, the Gabor transform, the use of Gabor multipliers creates a possibility to construct a time-variant filter. The Gabor transform is used to calculate time frequency coefficients, which are multiplied with a fixed time-frequency mask. Then the result is synthesized. If another way of calculating these coefficients is chosen or if another synthesis is used, many modifications can still be implemented as multipliers. For example, it seems quite natural to define wavelet multipliers. Therefore, for this case, it is quite natural to continue generalizing and look at multipliers with frames lacking any further structure.

Therefore, for Bessel sequences, the investigation of operators

M = ∑ mk < f , ψk > φk

where the analysis coefficients, < f , ψk >, are multiplied by a fixed symbol mk before resynthesis (with φk), is very natural and useful. These are the Bessel multipliers investigated in this project. The goal of this project is to set the mathematical basis to unify the approach to the Bessel multipliers for all possible analysis / synthesis sequences that form a Bessel sequence.

Bessel sequences and frames are used in many applications. They have the big advantage of allowing the possibility to interpret the analysis coefficients. This makes the formulation of a multiplier concept for other analysis / synthesis systems very profitable. One such system involves gamma tone filter banks, which are mainly used for analysis based on the auditory system.

- Balazs, P. (2007), "Basic Definition and Properties of Bessel Multipliers", Journal of Mathematical Analysis and Applications, 325, 1: 571--585. doi:10.1016/j.jmaa.2006.02.012, preprint

This project ended on 01.01.2007. Its completion allowed the sucessfull application for a 'High Potential'-Project of the WWTF, see MULAC.

Practical experience has shown that the concept of an orthonormal basis is not always useful. This led to the concept of frames. Models in physics and other application areas, including sound vibration analysis, are mostly continuous models. Many continuous model problems can be formulated as operator theory problems, as in differential or integral equations. An interesting class of operators is the Hilbert Schmidt class. This project aims to find the best approximation of any matrix by a frame multiplier, using the Hilbert Schmidt norm.

In finite dimensions, every sequence is a frame sequence, so the best approximation of any element can be found only via the frame operator using the dual frame for synthesis. Furthermore, the present best approximation algorithm involves the following steps: 1) The Hilbert-Schmidt inner product of the matrix and the projection operators involved is calculated in an efficient way; 2) Then the pseudo inverse of the Grame matrix is used to avoid the so-called calculation of the dual frames; The pseudo inverse is applied to the coefficients found above to find the lower symbol of the multiplier.

To find the best approximation of matrices via multipliers gives a way to find efficient algorithms to implement such operators. Any time-variant linear system can be modeled by a matrix. Time-invariant systems can be described as circulating matrices. Slowly-time-varying linear systems have a good chance at closely resembling Gabor multipliers. Other matrices can be well approximated by a "diagonalization" using other frames.

- P. Balazs, "Hilbert-Schmidt Operators and Frames - Classification, Approximation by Multipliers and Algorithms" , International Journal of Wavelets, Multiresolution and Information Processing, (2007, accepted) preprint, Codes and Pictures: here

This project ended on 01.01.2009. Its completion allowed the sucessfull application for a 'High Potential'-Project of the WWTF, see MULAC

HASSIP is a Research Training Network funded by the European Commission within the Improving the Human Potential program. The aim of the HASSIP network is to develop research activities and systematic interactions in mathematical analysis and statistics that are directly connected to signal and image processing. Although the Acoustics Research Institute was not initially a partner of this network, P. Balazs became a fellow of this network through cooperation with the group NuHAG.

- NuHAG, Faculty of Mathematics, University of Vienna
- Groupe de Traitement du Signal, Laboratoire d'Analyse Topologie et Probabilités, LATP/ CMI, Université de Provence, Marseille
- Modélisation, Synthèse et Contrôle des Signaux Sonores et Musicaux des LMA / CRNS Marseille
- Unité de physique théorique et de physique mathématique – FYMA

- Basic Properties of Bessel and Frame Multipliers: For Bessel sequences, the investigation of operators M = ∑ mk < f , ψk > is very natural and useful. The above M are Bessel multipliers. The goal of this project is to set the mathematical basis for this kind of operator.
- Best Approximation of Matrices by Frame Multipliers: Finding the best approximation by multipliers of matrices that represent time-variant systems gives a way to find efficient algorithms to implement such operators.

- P. Balazs, "Hilbert-Schmidt Operators and Frames - Classification, Approximation by Multipliers and Algorithms" , International Journal of Wavelets, Multiresolution and Information Processing, Vol. 6, No. 2, pp. 315 - 330, March 2008, preprint, Codes and Pictures: here
- P. Balazs, "Basic Definition and Properties of Bessel Multipliers", Journal of Mathematical Analysis and Applications, 325, 1: 571--585. (2007) doi:10.1016/j.jmaa.2006.02.012, preprint

This project ended on 01.01.2009. Its completion allowed the sucessfull application for a 'High Potential'-Project of the WWTF, see MULAC.

So-called Gabor multipliers are particular cases of time-variant filters. Recently, Gabor systems on irregular grids have become a popular research topic. This project deals with Gabor multipliers, as a specialization of frame multipliers on irregular grids.

The initial stage of this project aims to investigate the continuous dependence of an irregular Gabor multiplier on its parameter (i.e. the symbol), window, and lattice. Furthermore, an algorithm to find the best approximation of any matrix (i.e. any time-variant system) by such an irregular Gabor multiplier is being developed.

Gabor multipliers have been used implicitly for quite some time. Investigating the properties of these operators is a current topic for signal processing engineers. If the standard time-frequency grid is not useful to the application, it is natural to work with irregular grids. An example of this is the usage of non-linear frequency scales, like bark scales.

H. G. Feichtinger, NuHAG, Faculty of Mathematics, University of Vienna

This project ended on 28.02.2008 and is incorporated into the 'High Potential'-Project of the WWTF, MULAC (WWTF 2007).

Many problems in physics can be formulated as operator theory problems, such as in differential or integral equations. To function numerically, the operators must be discretized. One way to achieve discretization is to find (possibly infinite) matrices describing these operators using ONBs. In this project, we will use frames to investigate a way to describe an operator as a matrix.

The standard matrix description of operators O using an ONB (e_k) involves constructing a matrix M with the entries M_{j,k} = < O e_k, e_j>. In past publications, a concept that described operator R in a very similar way has been presented. However, this description of R used a frame and its canonical dual. Currently, a similar representation is being used for the description of operators using Gabor frames. In this project, we are going to develop and completely generalize this idea for Bessel sequences, frames, and Riesz sequences. We will also look at the dual function that assigns an operator to a matrix.

This "sampling of operators" is especially important for application areas where frames are heavily used, so that the link between model and discretization is maintained. To facilitate implementations, operator equations can be transformed into a finite, discrete problem with the finite section method (much in the same way as in the ONB case).

- P. Balazs, "Matrix Representation of Operators Using Frames", Sampling Theory in Signal and Image Processing (STSIP) (2007, accepted), preprint
- P. Balazs, "Hilbert-Schmidt Operators and Frames - Classification, Approximation by Multipliers and Algorithms" , International Journal of Wavelets, Multiresolution and Information Processing, (2007, accepted) preprint, Codes and Pictures: here

Weighted and controlled frames were introduced to speed up the inversion algorithm for the frame matrix of a wavelet frame. In this project, these kinds of frames are investigated further.

The frame multiplier concept is closely linked to the weighted frames concept. The frame operator of the weighted frame is simply a frame multiplier of the original frame. This project aims to explore this synergy. Finding an efficient way to invert the frame operator by applying weights to a given frame would be especially interesting. Weights are searched, for which the frame bounds are as "tight" as possible, meaning the spectrum is more concentrated. The first numerical experiments to find optimal weights have been conducted.

Weighted frames have already been applied to wavelets on the sphere. Also, the original work by Duffin and Schaefer dealt with the problem of finding such weights for a sequence of exponentials.

- J. P. Antoine, Unité de physique théorique et de physique mathématique – FYMA

In recent years, frames in signal processing applications have received more and more attention. Models, data, and operators must be discretized in order to function numerically. As a result, applications and algorithms always work with finite dimensional data. In the finite dimensional case, frames are equivalent to a spanning system. If reconstruction is wanted, frames are the only feasible generalization of bases. In contrast to bases, frames lose their linear independency. This project aims to investigate the properties of frames in finite dimensional spaces.

In this project, we will implement algorithms to work with frames in finite dimensional spaces. We will look at a way to "switch" between different frames, i.e. find a way to bijectively map between their coefficient spaces and provide the corresponding algorithm. This will be done by using the Cross-Gram matrix of the two involved frames. This matrix is a canonical extension of the basic transformation matrix used for orthonormal bases (ONB). The properties of the Gram matrix use a frame and its dual. We will investigate a criterion for finite dimensional spaces using frames. In particular, a space is finite dimensional if and only if Σ||g_{k}||^{2} < ∞.

Any analysis / synthesis system that allows perfect reconstruction is equivalent to a frame in its discrete version. This can be applied to Gabor, wavelet, or any other such system (e.g. a Gamma tone filter bank).

- P. Balazs,
*"Frames and Finite Dimensionality: Frame Transformation, Classification and Algorithms"*, submitted (2006), preprint - P. Balazs, "Matrix Representation of Operators Using Frames", Sampling Theory in Signal and Image Processing (STSIP) (2007, accepted), preprint

Practical experience quickly revealed that the concept of an orthonormal basis is not always useful. This led to the concept of frames. Models in physics and other application areas (for example sound vibration analysis) are mostly continuous models. Many continuous model problems can be formulated as operator theory problems, such as in differential or integral equations. Operators provide an opportunity to describe scientific models, and frames provide a way to discretize them.

Sequences are often used in physical models, allowing numerically unstable re- synthesis. This can be called an "unbounded frame". How this inversion can be regularized is being investigated. For many applications, a certain frame is very useful in describing the model. Therefore, it is also beneficial to use the same sequence to find a discretization of involved operators.

In this project, the theory of frames in the finite discrete case is investigated further.

The standard matrix description of operators using orthonormal bases is extended to the more general case of frames.

Weighted and controlled frames were introduced to speed up the inversion algorithm for the frame matrix of a wavelet frame. In this project, these kinds of frames are investigated further.

In this project, one function's sequences of irregular shifts are investigated.

- S. Heineken, Research Group on Real and Harmonic Analysis, University of Buenos Aires
- J. P. Antoine, Unité de physique théorique et de physique mathématique – FYMA
- M. El-Gebeily, Department of Mathematical Sciences, King Fahd University of Petroleum and Minerals, Saudi Arabia

This project ended in September 2011.

The** final MulAc Meeting** was in Vienna from 29th to 30th of August 2011.

The **ARI Mulac Frame Meeting** was held on Tuesday, June 15^{th} 2010at ARI.

The **MULAC Mid-term Meeting** was held in Marseille from 12. to 13. April 2010. See the Registration-Webpage or the Program.

The **FYMA Mulac semina**r was held in Louvain-la-Neuve in the 11th of March, 2010. (Talks by Jean-Pierre Antoine, Jean-Pierre Gazeau, Diana Stoeva and Peter Balazs.)

The MULAC - Kick-Off Meeting took place at ARI in Vienna from September 23^{rd} to 24^{th} 2008.

This international, multi-disciplinary and team-oriented project allowed P. Balazs to form a small group '*Mathematics and Acoustical Signal Processing*’ at the Acoustic Research Institute in cooperation with *NuHAG* Vienna (Hans G. Feichtinger),* LMA * (Richard Kronland-Martinet) and *LATP* Marseille (Bruno Torrésani) as well as the *FYMA* Louvain-la-Neuve (Jean-Pierre Antoine).

Within the institute the groups 'Audiological Acoustics' and 'Software Development' are involved.

This project is funded by the WWTF . It will run for 3,5 years and post-docs will be employed for six years total, as well as master students for 36 months total.

In December 2007 the Austrian Academy of Sciences was presenting '**mathematics in ...**' as the topic of the month . This included 'mathematics at the Acoustics Research Institute', which describes this project.

*"Frame Multipliers”* are a promising mathematical concept, which can be applied to retrieve desired information out of acoustic signals. P. Balazs introduced them by successfully generalizing existing time-variant filter approaches. This project aims to establish new results in the mathematical theory of frame multipliers, to integrate them in efficient digital signal processing algorithms and to make them available for use in 'real-world' acoustical applications. A multi-disciplinary and international cooperation has been established and will be extended in the project to create new significant impulses for the involved disciplines: mathematics, numerics, engineering, physics and cognitive sciences. Various acoustical applications like modelling of auditory perception, measurement of sound absorption coefficients and system identification of the head related transfer functions are included. The results of the project will allow their future integration into practical areas such as audio coding, noise abatement, sound quality design, virtual reality and hearing aids.

Weiterlesen: Frame Multiplier: Theory and Application in Acoustics (WWTF 2007)