Building Blocks

cysap believes that four building blocks are essential in any security analytics solution. They are,

1. Security Data Warehouse

Security Data warehouse serves as a central repository of all machine data collected in an organisation. It is the data store necessary for performing data mining and predictive analytics tasks.

Machine data collected in such data warehouse, along with knowledge management data ( data such as asset criticality, IP book, predictive analytic models etc. that are stored in traditional RDBMS tables) and any external feed (such as threat intelligence, IP geolocation etc) form the full set of data we will deal with while performing any analysis.

Technologies such as Hadoop with Hive and IBM Netezza are used as the core platform for building such security data warehouses.

Scalability and cost are the key factors to consider while building a data warehouse. Integration and support with analysis and modeling software or availability of any native analysis software are other key considerations. For instance below are examples of technologies used in building data warehouse and the statistical interfaces they have,

  • Hadoop and Hive interface with R via RHive or
  • IBM Netezza + SAS analysis interface
Situational awareness is realized via story boards ( similar to dashboards, but with plenty of visualizations and interactive features ).Organisations could perform real time or near real time analytics on machine data, the results of which will be fed into the dashboard.

  • Events occuring in the environment
  • Attributes or characteristics of the IT landscape and
  • Behavioral patterns¬†

are tracked through this dashboard.

Such attributes, characteristics and behavior could be too many for any one individual to keep track of or to make sense out of. Role based access should help in such cases.Only relevant visualizations, alerts and other analytics outputs will be made visible or brought to the notice of dashboard users.

2. Situational Awareness Dashboard

3. Analytics Workbench

Analytics workbench is the set of analysis and visualization tools that are used to develop models based on historical data sets.Workbench will be interfacing with data warehouse extensively.Developing and storing models, developing custom visualizations are done using analytics work bench.

Tools such as SAS, R, SPSS could be used to analyse data and develop models. Visualization tools could include Tableau or D3js.

This component could be outsourced or some activities in this building block can be taken as a service from vendors ( who offer machine learning as a service )

Search and in depth analysis of data is achieved through this building block. Apart from regular text based search feature, advanced features could include visualization, relationship analysis, geospatial and temporal analysis of search results are key elements.

Index and Search technologies such as Solr, ElasticSearch or Splunk are technologies that form the core of this building block.

Kibana and Prelert are some of the search clients that provide such advanced search functionality.

Collaboration is another key component of investigation dashboard. Users would like to get other domain experts involved in an ongoing investigation. Collaboration features will help drive such team driven investigations.

4. Investigation workbench