Fraunhofer Institute for Open Communication Systems

As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive biases learned by larger models during pre-training, we introduce a triple loss combining language modeling, distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device study.

A non-linear classification technique based on Fisher's discriminant is proposed. The main ingredient is the kernel trick which allows the efficient computation of Fisher discriminant in feature space. The linear classification in feature space corresponds to a (powerful) non-linear decision function in input space. Large scale simulations demonstrate the competitiveness of our approach.

Due to the volume conduction multichannel electroencephalogram (EEG) recordings give a rather blurred image of brain activity. Therefore spatial filters are extremely useful in single-trial analysis in order to improve the signal-to-noise ratio. There are powerful methods from machine learning and signal processing that permit the optimization of spatio-temporal filters for each subject in a data dependent fashion beyond the fixed filters based on the sensor geometry, e.g., Laplacians. Here we elucidate the theoretical background of the common spatial pattern (CSP) algorithm, a popular method in brain-computer interface (BCD research. Apart from reviewing several variants of the basic algorithm, we reveal tricks of the trade for achieving a powerful CSP performance, briefly elaborate on theoretical aspects of CSP, and demonstrate the application of CSP-type preprocessing in our studies of the Berlin BCI (BBCI) project.

After building a classifier with modern tools of machine learning we typically have a black box at hand that is able to predict well for unseen data. Thus, we get an answer to the question what is the most likely label of a given unseen data point. However, most methods will provide no answer why the model predicted the particular label for a single instance and what features were most influential for that particular instance. The only method that is currently able to provide such explanations are decision trees. This paper proposes a procedure which (based on a set of assumptions) allows to explain the decisions of any classification method.

How to evaluate innovations, especially in the beginning of new product development, is a question constantly posed by academics, managers, and policymakers. One reason for this is that improved front-end decisions greatly affect company performance. To find the answers to this question, this review article analyzes scientific publications on innovation indicators published between 1980 and 2015. The objective of this article is to increase the understanding of the indicator landscape and to complement the various stages of the innovation process with relevant indicators. In doing so, this study categorizes the identified indicators into company-specific and contextual dimensions. Furthermore, this study analyzes the indicators in terms of their potential for ex-ante and ex-post evaluation and investigates the characteristics of relevant publications. The analysis finds that more process rather than product indicators exist in the literature. Current publications emphasize qualitative and indirect indicators but neglect indicators for the early stages of the innovation process. The review identifies 82 unique indicators to evaluate innovations including 26 indicators for the early stages. The results can help managers, researchers, and policymakers to better understand the innovation process and the indicator landscape. However, more concrete indicators are needed to improve front-end innovation decisions.

This literature review has focused on smart governance as an emerging domain of study that attracts significant scientific and policy attention. More specifically, this paper aims to provide more insight in the definitions of and relationships between smart governance and concepts such as smart and electronic government, in the context of smart cities. The literature review shows that smart government can be considered as a basis for developing smart governance, through the application of emergent information and communication technologies (ICT) for governing. Smart governance as the intelligent use of ICT to improve decision-making through better collaboration among different stakeholders, including government and citizens, can be strongly related to government approaches. In this case ICT-based tools, such as social media, and openness can be factors that increase citizen engagement and support the development of new governance models for smart government. Smart governance may also have an important role in smart city initiatives, which require complex interactions between governments, citizens and other stakeholders. Based on the literature review, this paper coins a definition of ‘smart city governance’ and contributes to developing a framework for building new, smart governance models addressing the challenges of the digital society, collaborative governance, information sharing, citizen engagement, transparency and openness.

MOTIVATION: In order to extract protein sequences from nucleotide sequences, it is an important step to recognize points at which regions start that code for proteins. These points are called translation initiation sites (TIS). RESULTS: The task of finding TIS can be modeled as a classification problem. We demonstrate the applicability of support vector machines for this task, and show how to incorporate prior biological knowledge by engineering an appropriate kernel function. With the described techniques the recognition performance can be improved by 26% over leading existing approaches. We provide evidence that existing related methods (e.g. ESTScan) could profit from advanced TIS recognition.

A statistical theory for overtraining is proposed. The analysis treats general realizable stochastic neural networks, trained with Kullback-Leibler divergence in the asymptotic case of a large number of training examples. It is shown that the asymptotic gain in the generalization error is small if we perform early stopping, even if we have access to the optimal stopping time. Based on the cross-validation stopping we consider the ratio the examples should be divided into training and cross-validation sets in order to obtain the optimum performance. Although cross-validated early stopping is useless in the asymptotic region, it surely decreases the generalization error in the nonasymptotic region. Our large scale simulations done on a CM5 are in good agreement with our analytical findings.

This study analyses the impact of formal standards and regulation on firms’ innovation efficiency, considering different levels of market uncertainty. We argue that formal standards and regulation have different effects, depending on the extent of market uncertainty derived from theoretical considerations about information asymmetry and regulatory capture. Our empirical analysis is based on the German Community Innovation Survey (CIS). The results show that formal standards lead to lower innovation efficiency in markets with low uncertainty, while regulations have the opposite effect. In cases of high market uncertainty, we observe that regulation leads to lower innovation efficiency, while formal standards have the reverse effect. Our results have important implications for the future application of both instruments, showing that their benefits heavily depend on the market environment.

Brain-computer interfacing (BCI) is a steadily growing area of research. While initially BCI research was focused on applications for paralyzed patients, increasingly more alternative applications in healthy human subjects are proposed and investigated. In particular, monitoring of mental states and decoding of covert user states have seen a strong rise of interest. Here, we present some examples of such novel applications which provide evidence for the promising potential of BCI technology for non-medical uses. Furthermore, we discuss distinct methodological improvements required to bring non-medical applications of BCI technology to a diversity of layperson target groups, e.g., ease of use, minimal training, general usability, short control latencies.

The design and simulation of coding schemes, medium access control (MAC), and link-layer protocols for future industrial wireless local area networks can be supported by some understanding of the statistical properties of the bit error patterns delivered by a wireless link (which is an ensemble of transmitter, channel, receiver, modems). The authors present results of bit error measurements taken with an IEEE 802.11-compliant radio modem in an industrial environment. In addition to reporting the most important results, they draw some conclusions for the design of MAC and link-layer protocols. Furthermore, they show that the popular Gilbert/Elliot model and a modified version of it are a useful tool for simulating bit errors on a wireless link, despite their simplicity and failure to match certain measured statistics.

Incremental Support Vector Machines (SVM) are instrumental in practical applications of online learning. This work focuses on the design and analysis of efficient incremental SVM learning, with the aim of providing a fast, numerically stable and robust implementation. A detailed analysis of convergence and of algorithmic complexity of incremental SVM learning is carried out. Based on this analysis, a new design of storage and numerical operations is proposed, which speeds up the training of an incremental SVM by a factor of 5 to 20. The performance of the new algorithm is demonstrated in two scenarios: learning with limited resources and active learning. Various applications of the algorithm, such as in drug discovery, online monitoring of industrial devices and and surveillance of network traffic, can be foreseen.

Service function chaining is a network capability that provides support for application-driven-networking through the ordered interconnection of service functions. The lifecycle management of service function chains is enabled by two recently emerged technologies, software defined networking and network function virtualization, that promise a number of efficiency, effectiveness, and flexibility gains. This article introduces a service function chaining taxonomy that considers architecture and performance dimensions as the basis for the subsequent state-of- the-art analysis. The article concludes with a gap analysis of existing solutions and the identification of future research challenges.

This paper proposes a Green Light Optimized Speed Advisory (GLOSA) application implementation in a typical reference area, and presents the results of its performance analysis using an integrated cooperative ITS simulation platform. Our interest was to monitor the impacts of GLOSA on fuel and traffic efficiency by introducing metrics for average fuel consumption and average stop time behind a traffic light, respectively. For gathering the results we implemented a traffic scenario defining a single route through an urban area including two traffic lights. The simulations are varied for different penetration rates of GLOSA-equipped vehicles and traffic density. Our results indicate that GLOSA systems could improve fuel consumption and reduce traffic congestion in junctions.

A new efficient algorithm is presented for joint diagonalization of several matrices. The algorithm is based on the Frobenius-norm formulation of the joint diagonalization problem, and addresses diagonalization with a general, non-orthogonal transformation. The iterative scheme of the algorithm is based on a multiplicative update which ensures the invertibility of the diagonalizer. The algorithm 's efficiency stems from the special approximation of the cost function resulting in a sparse, block-diagonal Hessian to be used in the computation of the quasi-Newton update step. Extensive numerical simulations illustrate the performance of the algorithm and provide a comparison to other leading diagonalization methods. The results of such comparison demonstrate that the proposed algorithm is a viable alternative to existing state-of-the-art joint diagonalization algorithms.

In this paper, we identify the strategic motives of German manufacturing companies in the electrical engineering and machinery industry to be involved in standards development organizations. First, we present the general motives for the formation of strategic alliances and relate them to specific standardization motives. Then, we identify pursuing specific company interests, solving technical problems, knowledge seeking, influencing regulation, and facilitating market access as motives to standardize by means of factor analysis. In a second step, we test hypotheses on the relationship between the importance of strategic motives and firm level variables, e.g. R&D intensity, innovation activities, and firm size. The results reveal that firms in electric engineering and machinery have a particularly strong interest in ensuring industry-friendly design of regulations, which can be achieved by standards. Moreover, the results confirm that small firms also from these two sectors are active in standardization alliances to access knowledge from other involved stakeholders.

The open architecture of the Internet and the use of open standards like Session Initiation Protocol (SIP) constitute the provisioning of services (e.g., Internet telephony, instant messaging, presence, etc.) vulnerable to known Internet attacks, while at the same time introducing new security problems based on these standards that cannot been tackled with current security mechanisms. This article identifies and describes security problems in the SIP protocol that may lead to denial of service. Such security problems include flooding attacks, security vulnerabilities in parser implementations, and attacks exploiting vulnerabilities at the signaling-application level. A qualitative analysis of these security flaws and their impacts on SIP systems is presented.

The objective of this article is to demonstrate the feasibility of on-demand creation of cloud-based elastic mobile core networks, along with their lifecycle management. For this purpose the article describes the key elements to realize the architectural vision of EPC as a Service, an implementation option of the Evolved Packet Core, as specified by 3GPP, which can be deployed in cloud environments. To meet several challenging requirements associated with the implementation of EPC over a cloud infrastructure and providing it "as a Service," this article presents a number of different options, each with different characteristics, advantages, and disadvantages. A thorough analysis comparing the different implementation options is also presented.

A principal objective of this document is to describe the underlying framework of middlebox communications (MIDCOM) to enable complex applications through the middleboxes, seamlessly using a trusted third party.This document and a companion document on MIDCOM requirements ([REQMTS]) have been created as a precursor to rechartering the MIDCOM working group.There are a variety of intermediate devices in the Internet today that require application intelligence for their operation.Datagrams pertaining to real-time streaming applications, such as SIP and H.323, and peer-to-peer applications, such as Napster and NetMeeting, cannot be identified by merely examining packet headers.Middleboxes implementing Firewall and Network Address Translator services typically embed application intelligence within the device for their operation.The document specifies an architecture and framework in which trusted third parties can be delegated to assist the middleboxes to perform their operation, without resorting to embedding application intelligence.Doing this will allow a middlebox to continue to provide the services, while keeping the middlebox application agnostic.

In this article we address the issue of denial of service attacks targeting the hardware and software of voice over IP servers or by misusing specific signaling protocol features. As a signaling protocol we investigate here the session initiation protocol. In this context we mainly identify attacks based on exhaustion of the memory of VoIP servers, or attacks that incur high CPU load. We deliver an overview of different attack possibilities and explain some attacks in more detail, including attacks utilizing the DNS system and those targeting the parser. A major conclusion of the work is the knowledge that SIP provides a wide range of features that can be used to mount DoS attacks. Discovering these attacks is inherently difficult, as is the case with DoS attacks on other IP components. However, with adequate server design, efficient implementation, and appropriate hardware, the effects of a large portion of attacks can be reduced