Mathematical Analysis and Improvement of the Maximum Spatial Eigenfilter for Direction of Arrival Estimation

Maximum spatial eigenfiltering improves the accuracy of maximum likelihood direction-of-arrival estimators for closelyspaced signal sources but may interchangeably attenuate widelyspaced signal sources, producing a severe performance degradation. Although this behavior has been observed experimentally, it still lacks a mathematical explanation. In our previous work, we overcame these limitations using a differential spectrum-based spatial filter but this still caused a small degradation in the DOA estimate. In this paper, we develop a mathematical analysis of how the signal source separation and the Karhunen-Loève expansion affect the passbands of the maximum spatial eigenfilter. The farther the sources, the less significant is the maximum eigenvalue of the spatial correlation matrix and its corresponding eigenvector. Then, the magnitude response of the maximum spatial eigenfilter no longer approximates the spatial power spectrum and is not guaranteed to place multiple passbands around the signal sources. Consequently, we propose a spatial filter built from the eigenvectors of the entire signal subspace. This filter showed an overall runtime smaller than that of our previous work. It also provides a significant reduction in the threshold signal-to-noise ratio for closely-spaced signal sources and does not hamper the estimation for widely-spaced signal sources.


I. INTRODUCTION
Localization techniques enable several types of state-of-the-art commercial and government applications and services [1]. These techniques are regarded as a core technology in fourth and fifth generation (4G/5G) wireless telecommunications systems as they allow forming beams to individual mobile users in a multiple-input multiple-output (MIMO) urban environment [2], increasing spectral efficiency and the capacity of those systems [3]. They are also useful to estimate the position of indoor wireless users [1], for target localization and imaging using MIMO radars [4], and even for localization of long-distance underwater acoustic sources [5].
In this context, direction-of-arrival (DOA) estimation plays a key role, since all these applications need to know a priori the signal source position. Currently, there are several DOA estimators but the most representative ones are: Multiple Signal Classification (MUSIC) [6], Estimation of Signal Mathematical Analysis and Improvement of the Maximum Spatial Eigenfilter for Direction of Arrival Estimation source, regardless of whether they are closely or widely-spaced from each other.
This work is organized as follows: Section II sets up the signal model for DOA estimation, provides an overview on the conventional Modified MODEX, and depicts the maximum spatial eigenfilter.
Section III analyses the filter behavior, and Section IV presents our new proposition. Section V brings the experimental results, and Section VI draws conclusions and suggestions for future works.

II. THEORETICAL FRAMEWORK
Let us consider narrowband far-field signal sources impinging at DOAs , = 1, …, , on a uniform linear array (ULA) formed by half-wavelength-spaced sensors, with > . The set of snapshots of the array output is modeled as [13]: where are its eigenvalues arranged in descending order of magnitude and are their corresponding eigenvectors. The first = min , rank eigenvalues along with 1 , …, span the signal subspace of . The remaining − eigenvalues and eigenvectors span its orthogonal or noise subspace [24].
The conventional Modified MODEX uses MODE to compute three solutions to the DOA problem in order to keep good asymptotic estimation performance [14]. The first one is calculated from the complete MODE algorithm, estimating the roots 1 , …, of the polynomial = 0 + 1 −1 + ⋯ + by solving the following non-linear optimization problem: where = 0 , …, contains the polynomial coefficients and is subject to the unit-norm constraint ≠ to avoid a trivial solution, and to the conjugate-symmetry constraint = − * , = 1, …, , where ⋅ * is the complex conjugate operator [14]; = 1 , …, ; +1 , …, and: such that H = . The other two solutions come respectively from the imposition of the real linear constraint ℜ 0 = 1 and imaginary linear constraint ℑ = 1 , both computed using the first solution as a parameter.
However, this procedure generates 3 estimates. Thus, an ML procedure must take place to combine those estimates into -tuples that are included in the set to select the best one among them: where is a candidate matrix of steering vectors for each -tuple in .
Even though the noise will not affect distinct MODE solutions simultaneously and/or the same estimate in each solution [14], the threshold breakdown effect may occur as the SNR gets below a critical value [15]. Then, the noise present in worsens the ML procedure performance in Eq. (5) such that the conventional Modified MODEX may not choose the best M-tuple [15].
Since the maximum eigenfilter, i.e. the eigenvector associated with the largest eigenvalue of [17], maximizes the output SNR, Krummenauer et al. [15] proposed using it as a spatial filter [16] to reduce the noise influence in Eq.
where = ; = H and is the following + × convolution matrix: The coefficient vector = ℎ 0 , …, ℎ , 0 < < , corresponds to the maximum eigenvector of the matrix given by the average of the − submatrices ∈ ℂ +1 ×( +1) along the main diagonal of [15]: Then, we can find from the eigendecomposition of : where the eigenvalues are arranged in descending order of magnitude and are their corresponding eigenvectors. In other words, the maximum spatial eigenfilter is simply given by = 1 [15].

III. PROBLEM STATEMENT AND DIAGNOSIS
We had already stated in [18] that, despite the optimality of this spatial eigenfilter in maximizing the SNR at its output, its magnitude response may interchangeably attenuate signal sources widelyspaced from the others even in absence of noise. In such cases, the filter may cause Modified MODEX to take a wrong decision in the ML selection procedure, yielding a severe degradation in estimation performance. Although this unexpected behavior has been observed experimentally, it still lacks an analytical explanation.
To derive a mathematical reasoning about the selectivity of that filter, we must bring our discussion to the frequency domain. Then, the magnitude response of the maximum spatial eigenfilter is given by: where ,1 is the -th element of 1 and = sin . To assess the effectiveness of , we first estimate the spatial power spectrum of the signals impinging on the array by taking the Fourier transform of the first column of as: where , 1 is its -th element and is the estimated spatial correlation function at lag .
According to [19]- [21], the difference between 1 and 2 becomes larger whenever signal sources are more closely spaced. In this case, 1 ≫ 2 so that 1 holds almost all signal energy, as shown by the upper dashed line in Fig. 1 where , 1 ≤ ≤ + 1 , corresponds to the -th column of .
Therefore, = 1 ≈ 1 / 1 1,1 * such that corresponds to the spatial power spectrum calculated using only + 1 samples: As a consequence, the filter passband encompasses both DOAs of the signal sources, what explains why the maximum spatial eigenfilter performed so well for 1 , 2 = 10 ∘ , 15 ∘ in [15]. On the other hand, this is not true for widely-spaced signal sources. In this case, 1 and 2 get close to each other such that 1 holds only half of signal energy, as indicated by the lower dashed line in Fig. 1 for 1 , 2 = 10 ∘ , 45 ∘ . So, Eq. (12) no longer holds, since the first column of depends more heavily on 2 , …, +1 : Then, the filter coefficient vector becomes rather different from the spatial correlation vector and the magnitude response of the maximum spatial eigenfilter may not sufficiently resemble . Consequently, is not guaranteed to place multiple passbands around widely-spaced signal sources. Instead, the filter may interchangeably attenuate them even in absence of noise, as shown in Fig. 3a and 3b for 1 , 2 = 10 ∘ , 45 ∘ .
In both cases, Fig. 1 shows that the percentage of signal energy in 1 gets smaller as the SNR decreases. So, the maximum eigenvector 1 becomes even less suitable to approximate 1 such that the maximum spatial eigenfilter may not be able to preserve all signal sources at low SNRs.
IV. MAXIMIZING THE OUTPUT SIGNAL POWER WITH A SIGNAL SUBSPACE FILTER The observations presented in the last section lead us to conclude that the maximum spatial eigenfilter may, in principle, sacrifice some of the signal sources to maximize the SNR at its output, causing Modified MODEX to make an incorrect decision in the ML procedure.
In [18], we introduced a Moving Average eigenvalue-based multiband spatial filter computed from the differential spectrum. That filter is able to place passbands on the DOAs of the signal sources at the expense of performing multiple eigendecompositions of . However, to avoid a prohibitive growth in runtime, we had to reduce both the spectral resolution and filter order, what weakened its ability to attenuate noise outside the passbands. Then, for widely-spaced signal sources, even though that filter performed significantly better than the maximum spatial eigenfilter, it still performed slightly worse than the conventional Modified MODEX.
Following from Eq. (14), if a spatial filter whose coefficient vector was made dependent on 1 , …, +1 in the same way as 1 : then its magnitude response would be supposed to place multiple passbands around the signal sources regardless of whether they are closely or widely spaced. Additionally, this eigenvector-based multiband filter would be faster to calculate than that differential spectrum-based filter. So, in order to preserve all signal sources at the filter output and reduce the noise influence on the filter calculation, we propose to derive a finite impulse response (FIR) filter that maximizes the output SNR subject to its coefficient vector being dependent on the eigenvectors of only the signal subspace of .
According to [26], the output SNR of a FIR filter is given by: So, the optimization of the filter coefficient vector can be written as: Since 2 does not influence the maximization, the problem simply becomes: Then, if we replace with Eq. (9), the objective function can be written as: where ⋅ , ⋅ stands for the inner product. But the coefficient vector that maximizes Eq. (19) is the same that maximizes each of its addends, so: Since is not a function of , we can do: such that the expression inside curly brackets is the cosine similarity between and . Now, by imposing the constraint, Eq. (21) becomes: Considering the cosine similarity is maximum for identical vectors, we conclude that: Pre-multiplying Eq. (23) with H , we get: As shown in Fig. 4a, the magnitude responses are complementary and their combination place passbands around the actual DOAs. However, the corresponding phase responses ∠ do not match, as can be seen in Fig. 4b. So, before combining the filters, we can equalize the phase response of each one of them by simply convolving its impulse response with its corresponding timereversed, time-shifted, conjugated version, in order to avoid magnitude and phase distortion. Then, this allows us to derive a filter of order 2 whose coefficient vector is given by the following weighted sum: where conv ⋅ stands for the convolution operator, and is the + 1 × + 1 co-identity matrix [25]. In frequency domain, the frequency response of this linear phase filter corresponds to: In order to evaluate the effectiveness of this filter, we applied it to the cases 1 , 2 = 10 ∘ , 15 ∘ and 1 , 2 = 10 ∘ , 45 ∘ , as shown respectively in Fig. 5a and 5b. The magnitude response of the proposed filter was able to place multiple passbands around the DOAs of the signal sources, preserving the power of both signals regardless of their spacing.
However, using the entire signal subspace of increases the probability of subspace swap [26], [27]. So, spurious peaks are more likely to arise in the filter magnitude response for closely-spaced signal sources at low SNR values. This may compromise the ML procedure and cause performance degradation under severe noise conditions [28].

V. RESULTS
In this section, we compare the estimation performance of Modified MODEX using our present proposition during the ML procedure to those using the maximum spatial eigenfilter [15], and the differential spectrum-based filter [18].
To do so, we performed = 1000 Monte-Carlo experiments for each SNR ranging from −15 to 15 dB in steps of 1.25 dB, considering a ULA composed of = 10 sensors that take = 100 snapshots of = 2 narrowband signal sources. We adopted order = 7 as in [15] for all filters evaluated in this work. Additionally, the differential spectrum was calculated using 32 samples. Since the filters were evaluated according to the signal source separation, we considered one signal source Additionally, the case where both signal sources have 1 = 2 = 10 ∘ was unconsidered.
The estimation performance was evaluated using Root Mean Square Error (RMSE) for each SNR value. RMSE is compared to Cramér-Rao Lower Bound (CRLB) [29] and the threshold SNR is discussed.
Figs. 6 to 8 respectively show gray-shaded RMSE surfaces for Modified MODEX using the maximum spatial eigenfilter, the differential spectrum-based filter, and the filter proposed in this work.
The maximum spatial eigenfilter significantly improved the estimation performance for closelyspaced signal sources such that its RMSE surface was below the CRLB, as presented in Fig. 6. The threshold SNR for 1 , 2 = 10 ∘ , 15 ∘ was −12.50 dB. However, the maximum spatial eigenfilter caused a severe degradation on estimation performance for widely-spaced signal sources.
The differential spectrum-based filter performed a little worse than the maximum spatial eigenfilter for closely-spaced signal sources, as shown in Fig. 7. For instance, its threshold SNR was −11.25 dB for 1 , 2 = 10 ∘ , 15 ∘ . On the other hand, it significantly improved the estimation performance for widely-spaced signal sources compared to the maximum spatial eigenfilter. For Finally, Fig. 8 shows that our proposition performed similarly to the other two filters for closelyspaced signal sources. However, it is worth noting that the RMSE surface attained the CRLB and the threshold SNR was mostly kept at −10 dB regardless of the signal source separation. For instance, it achieved −10 dB for both 1 , 2 = 10 ∘ , 15 ∘ and 1 , 2 = 10 ∘ , 45 ∘ . Therefore, the proposed spatial filter in no way hampered the estimation performance of the conventional Modified MODEX and significantly improved it for closely-spaced signal sources, keeping the threshold SNR at −10 dB for almost any signal source separation.
As a total of one million experiments of Modified MODEX were carried out to calculate the RMSE surfaces for each filter, we took this opportunity to record their respective computation times.
Regarding the computational effort, filtering techniques clearly increase the total computation time, since additional calculations are needed. Then, we measured the runtime of each experiment for Modified MODEX taking into account both the DOA estimation time and filter calculation time when milliseconds (ms) per experiment. Modified MODEX using maximum spatial eigenfiltering took 3.49 ms on average. With the differential spectrum-based filter, this time was 4.11 ms, and using our present proposition, the mean runtime was 3.62 ms. In relative terms, Modified MODEX using our present proposition was 13% slower than the conventional version, 4% slower than using the maximum spatial eigenfilter but 12% faster than using the differential spectrum-based filter.
We also must note that our proposition makes use of the convolution operation -Eq. (27) -and, with = 7 , its impulse response vector has 15 coefficients instead of only 8 for the other filters.
This causes to assume larger dimensions and, consequently, the amount of time spent to calculate Eq. (6) in the ML procedure is larger. Even so, our proposition performed faster than the differential spectrum-based filter, while delivering better results.

VI. CONCLUSIONS AND REMARKS
Although the maximum spatial eigenfilter allowed a significant improvement in the DOA estimation of closely-spaced signal sources, we observed that its magnitude response may interchangeably attenuate widely-spaced signal sources, leading to a severe performance degradation and increasing the threshold SNR. We pointed that out firstly in [18], but without an accompanying analytical explanation. At that time, we proposed the differential spectrum-based filter that, despite largely overcoming the limitations of the maximum spatial eigenfilter, it still caused a slightly increase in the threshold SNR for widely-spaced signal sources.
From Karhunen-Loève expansion, we mathematically explained this unexpected behavior in terms of the relation between the energy of the principal eigenvalues of the spatial correlation matrix and the separation between the signal sources, as pointed out in [21]. We proved that, for closely-spaced signal sources, the maximum eigenvalue concentrates almost all signal energy and the coefficients of the maximum spatial eigenfilter approximate the signal spatial correlation vector. In this case, the filter magnitude response places passbands around the DOAs of the signal sources. On the other hand, for widely-spaced signal sources, half of signal energy spreads through the signal subspace such that the maximum spatial eigenfilter no longer approximates the spatial power spectrum and can even suppress any of the signal sources.
As a corollary, to circumvent that problem, we derived a new multiband filter that maximizes signal power at its output by combining the eigenvectors of the entire signal subspace. To avoid phase distortion, we performed a phase equalization procedure and the resulting filter was able to place multiple passbands around the DOAs of all signal sources, regardless of their separation.
Experimental results showed that using our proposition during the ML procedure in no way hampered the estimation performance of the conventional Modified MODEX and significantly improved it for closely-spaced signal sources. In addition, it kept the threshold SNR almost independent of signal source separation at the expense of a small increase in runtime.
As future work, we plan extending the scope of spatial filtering to other important DOA estimation methods beyond MODEX-based estimators. In addition, we intend to evaluate the distortion effects caused by the non-flat gain of the filter passbands on the DOA estimation in two or three dimensions for wideband systems.