Do you know the basic Search Experiences / Features supported as part of core Foundational Search Capability? Let’s take a look!…
Foundation Principle – it supports Query-based search functionality. This is a basic search functionality should work as expected, like a query. Search functionality is supported by three mandatory features/values.
- Scale – supports runs on huge large data across the cluster and gets you all results with broader data
- Speed – search gets executed in few milliseconds
- Relevance – search gives only relevance data more appropriate to your query
- Visualization – You can use UI to slide left/right/ and narrow-down search results
- Filtering – You can filter large data sets with some criteria on data elements in your search results
Additional Search Features
Search leverages data and index Cross Cluster Replication; it also supports Hot/Warm/Cold data Architecture, it leverages Index Lifecycle Mgmt, Frozen Data, etc. Data what gets stored is bounded based on Schema, and Documents, etc.
Logstash is an open-source, server-side data processing pipeline that ingests data from a different source of an enterprise. Data is organized and distributed from many systems in many formats. Logstash supports a variety of inputs that pull in events from a multitude of common sources, all at the same time. It easily ingests from various logs, metrics, web services, data repositories, and various cloud services in a continue streaming fashion. Generally, data travels from source to store, Logstash filters parse each incident, identify various named fields to build structure, and transform them to converge on a common pre-defined schema format for more powerful analysis and business value.
Kibana provides you an opportunity to select the way you want to present your data. You may not need to know what you’re looking for, with its interactive visualization capabilities and features, you can define elements what you want to see in the User Interface. Kibana core functionality delivers with the classic histograms, line graphs, pie charts, sunbursts, etc. You can use Vega grammar to design your visualizations. All leverage the full aggregation capabilities of Elasticsearch. Kibana makes you visualize your Elasticsearch data and navigate the Elastic Stack.
Beats deliver the data shipper functionality. Beats are data shippers sends data from thousands of machines and systems to Logstash or Elasticsearch. Beats are suitable for gathering data, and they run on your servers with your containers, or deploy as serverless functions, and centralize data in Elasticsearch. Beats can also ship to Logstash for transformation and parsing. There are various beats like Filebeat using Log Files, Metricbeats from Metrics, Packetbeat using Network Data, Winlogbeat from Windows Event Logs, Auditbeat from Audit Data, Heartbeat from Uptime Monitoring, Functionbeat from Serverless Functions.
Offering / Delivery Model
The following are different options of how elastic search can be setup.
Foundational Open Source Core – This is considered as a base and set of default core services included, all these are open-sourced and free to use. This includes Logstash, Beats, Elastic Search, Kibana, Elastic Search Core Security (ESS) and Elastic Cloud on Kubernetes (ECK). These come with both you install and manage and in the form of managed service as well.
Free Cloud Subscription – Elastic has collaboration with AWS and Google, as part of managed services offering, there is a free tier offering from the elastic cloud on kubernetes, the default distribution of ECK is free, all the source code for ECK publicly viewable, licensed under the Elastic License.
Enterprise Cloud subscription – Enables additional features in the cloud, including the ability to deploy clusters with advanced features such as field- and document-level access control, machine learning, graph analytics, and more. In the future, the Enterprise subscription will also unlock additional advanced orchestration features.
- Logging – Elastic Stack has one of the key and common solution is logging, it has the most popular open-source login platform. It consolidates logs from your application, servers, virtual machines, and containers and more and facilitate to deep dive and analyze in a single screen with a centralized view.
- Application Performance Monitoring – provides you to monitor software services and applications in real-time, collects various metrics like host level, agent level, JVM level and also it collects various unhandled errors and exceptions will help to debug performance level issues easy and faster.
- Metrics – Elastic Stack provides you a single dashboard view of all metrics you want to analyze in your application, you can continuously explore and various attributes, hostnames, addresses, tags, and create time-series data charts, slice, and dice various metrics.
- Uptime Monitoring – This supported Measuring the application availability and supported using constantly pinging ports, hosts, containers, and various URLs and endpoints (HTTP/TCP/ICMP) of an application using HeartBeat a lightweight component.
- Site Search – Service supported using a crawler ingesting into your website and will look for always new content in your website, and it can perform real-time or scheduled crawling and make your site searchable for your customers.
- App Search – Service backed by Elasticsearch, with relevance models optimized for real-life search. Take advantage of typo-tolerance, schema-less ingestion, and real-time data.
- Enterprise Search – Service supports like one-stop-shop to search data at Enterprise level like creating a single source of truth once you connect all your data sources of enterprise irrespective type of content, format or document, etc.
- security information and event management (SIEM) provides network and host data integrations, shareable analytics based on the Elastic Common Schema (ECS), and the ability to explore and audit your enterprise security data like Audit events, DNS traffic, authentication logs, etc
- Elastic Maps – Location is also a key in any part of transaction and data, so to supports “where” , Elastic Maps provides location monitoring and adds location intelligence, analytics on geo maps with visualization
- Business Analytics – dedicated solution supports on visualization and analytics on your financial data, business retail data, sales pipeline data, and your web site traffic data, etc.
Logging: Logging and Log Analysis is often considered the primary use case for Elasticsearch and the ELK stack. Any logs can be transported if they’re transformed into a JSON format. There are multiple tools than can be leveraged for importing logs. Some are ELK components such as Logstash and Beats. Others are non-ELK components like Kafka or custom applications using features like the java bulk loader.
Event Data and Metrics: This use case has data that is similar to log data in shape and volume. The key differences are Data Pertinence, where an event is distinguished from other log data based upon its relevance; Context, where an event must have enough context provided to make it understandable; and, Timeliness, where event data must be associated with a precise time. Based on these differences, event data is also visualized and analyzed differently from log data.
Full-Text Search: As Elastic says, Elasticsearch is for searching. Full-text search is useful for features like webform autocomplete, text proximity, simple and prefix term searches, and weighted term search queries. Within Kibana, the developer console allows for extensive use of Query DSL to tune query structure and performance.
Data Visualization: Data Visualization is a growing use case as Kibana develops. Graphs, heatmaps, and charts have long been supported and integrated into Dashboards. Now, spaces have been added to Kibana to provide user or group-specific content access, allowing easier and more focused dashboard and visualization sharing.
Intersys Consulting offers the experience and knowledge of a great data engineering team experts that can help you succeed on your data project, contact us!