Elasticsearch Setup & Operations

What is Elasticsearch?

Elasticsearch is a distributed search and analytics engine built on top of Apache Lucene. Key features:

Full-text search: Fast indexing and querying of large text datasets
Log analytics: Centralized log collection and analysis with Kibana visualization
Time-series data: Optimized for metrics and time-based analytics
Real-time indexing: Near-instant data availability after ingestion
Scalability: Horizontal scaling across multiple nodes and clusters
RESTful API: Simple HTTP interface for all operations

Installation on Ubuntu/Debian

Add Elastic Repository

# Install GPG key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -

# Add repository
apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | tee /etc/apt/sources.list.d/elastic-8.x.list

# Update package list
apt-get update

Install Elasticsearch

apt-get install elasticsearch

This installs the latest 8.x version with security features enabled by default.

Enable and Start Service

# Enable on boot
systemctl enable elasticsearch

# Start the service
systemctl start elasticsearch

# Check status
systemctl status elasticsearch

Verify Installation

# Check cluster health
curl -k -u elastic:YOUR_PASSWORD https://localhost:9200/_cluster/health?pretty

# Expected output:
# {
#   "cluster_name" : "elasticsearch",
#   "status" : "green",
#   "timed_out" : false,
#   "number_of_nodes" : 1,
#   "active_primary_shards" : 0
# }

Configuration

Edit /etc/elasticsearch/elasticsearch.yml:

# Cluster settings
cluster.name: my-cluster
node.name: es-node-01
node.roles: [master, data, ingest]

# Network settings (single-node setup)
network.host: 0.0.0.0
http.port: 9200
discovery.type: single-node

# Security settings
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true

# JVM heap size (set in jvm.options)
# Elasticsearch: Set to half of available RAM, max 31GB

# Cluster discovery (for multi-node)
# discovery.seed_hosts: ["es-node-01", "es-node-02", "es-node-03"]
# cluster.initial_master_nodes: ["es-node-01", "es-node-02", "es-node-03"]

Configure Heap Size

Edit /etc/elasticsearch/jvm.options.d/heap.options:

# Example: 4GB heap on 8GB system
-Xms4g
-Xmx4g

Never set heap size above 31GB due to JVM compressed object pointer limitations. For systems with more RAM, add additional Elasticsearch nodes instead of increasing heap.

Restart Elasticsearch:

systemctl restart elasticsearch

Basic Index Operations

Create an Index

curl -X PUT -k -u elastic:PASSWORD https://localhost:9200/my-index-001 \
  -H 'Content-Type: application/json' \
  -d '{
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0
    },
    "mappings": {
      "properties": {
        "timestamp": { "type": "date" },
        "message": { "type": "text" },
        "level": { "type": "keyword" },
        "host": { "type": "keyword" }
      }
    }
  }'

Index a Document

curl -X POST -k -u elastic:PASSWORD https://localhost:9200/my-index-001/_doc \
  -H 'Content-Type: application/json' \
  -d '{
    "timestamp": "2026-03-29T10:30:00Z",
    "message": "Application started successfully",
    "level": "INFO",
    "host": "app-server-01"
  }'

Search Documents

curl -X GET -k -u elastic:PASSWORD https://localhost:9200/my-index-001/_search \
  -H 'Content-Type: application/json' \
  -d '{
    "query": {
      "match": {
        "message": "started"
      }
    },
    "size": 10
  }'

Complex Query Example

curl -X GET -k -u elastic:PASSWORD https://localhost:9200/my-index-001/_search \
  -H 'Content-Type: application/json' \
  -d '{
    "query": {
      "bool": {
        "must": [
          { "match": { "message": "error" } },
          { "range": { "timestamp": { "gte": "2026-03-28T00:00:00Z" } } }
        ],
        "filter": [
          { "term": { "level": "ERROR" } }
        ]
      }
    },
    "size": 20,
    "sort": [
      { "timestamp": { "order": "desc" } }
    ]
  }'

Kibana Installation & Setup

Install Kibana

apt-get install kibana

Configure Kibana

Edit /etc/kibana/kibana.yml:

# Server settings
server.port: 5601
server.host: "0.0.0.0"
server.publicBaseUrl: "https://kibana.example.com"

# Elasticsearch connection
elasticsearch.hosts: ["https://localhost:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "KIBANA_PASSWORD"

# Security
xpack.security.enabled: true

# Logging
logging.level: info

Enable and Start Kibana

systemctl enable kibana
systemctl start kibana

Setup Enrollment Token

First time setup, generate an enrollment token on Elasticsearch:

/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana

Enter this token in the Kibana setup wizard at http://localhost:5601 (or your domain).

Create Index Pattern

Navigate to Management → Data Views
Click Create data view
Index pattern: my-index-*
Timestamp field: timestamp
Click Save data view

Log Rotation and Disk Management

Enable Index Lifecycle Management (ILM)

Create an ILM policy for log rotation:

curl -X PUT -k -u elastic:PASSWORD https://localhost:9200/_ilm/policy/logs-policy \
  -H 'Content-Type: application/json' \
  -d '{
    "policy": "logs-policy",
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb"
          }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "set_priority": {
            "priority": 50
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }'

Monitor Disk Usage

# Check cluster disk usage
curl -k -u elastic:PASSWORD https://localhost:9200/_cat/allocation?v

# Check index size
curl -k -u elastic:PASSWORD https://localhost:9200/_cat/indices?v&h=index,store.size,docs.count

Delete Old Indices Manually

# Delete indices older than 90 days
curl -X DELETE -k -u elastic:PASSWORD https://localhost:9200/logs-2025.12.*

Backup and Restore

Register Snapshot Repository

curl -X PUT -k -u elastic:PASSWORD https://localhost:9200/_snapshot/backup-repo \
  -H 'Content-Type: application/json' \
  -d '{
    "type": "fs",
    "settings": {
      "location": "/mnt/elasticsearch-backups"
    }
  }'

Create the directory:

mkdir -p /mnt/elasticsearch-backups
chown elasticsearch:elasticsearch /mnt/elasticsearch-backups
chmod 755 /mnt/elasticsearch-backups

Create Snapshot

curl -X PUT -k -u elastic:PASSWORD https://localhost:9200/_snapshot/backup-repo/snapshot-001

Restore from Snapshot

curl -X POST -k -u elastic:PASSWORD https://localhost:9200/_snapshot/backup-repo/snapshot-001/_restore

Monitoring and Alerting

Check Node Status

curl -k -u elastic:PASSWORD https://localhost:9200/_nodes/stats?pretty

Enable X-Pack Monitoring

Already enabled in Elasticsearch 8.x. View in Kibana under Management → Stack Monitoring.

Common Alerts to Configure

Cluster health: Alert when status is yellow or red
Disk space: Alert when available disk < 15%
JVM memory: Alert when heap usage > 85%
Unassigned shards: Alert when count > 0

Set up alerts in Kibana under Stack Management → Alerting.

Use OpenSearch (AWS-maintained Elasticsearch fork) as an alternative if you prefer:

Open-source governance
AWS-integrated features
Compatible API with Elasticsearch 7.10 base
Docker support: docker run opensearchproject/opensearch:latest

Performance Tuning

Refresh Interval

Reduce refresh frequency for bulk indexing:

curl -X PUT -k -u elastic:PASSWORD https://localhost:9200/my-index-001/_settings \
  -H 'Content-Type: application/json' \
  -d '{
    "settings": {
      "refresh_interval": "30s"
    }
  }'

Shard Allocation Awareness

For multi-node production clusters:

# In elasticsearch.yml
node.attr.zone: "zone-a"
cluster.routing.allocation.awareness.attributes: zone

Bulk Indexing Optimization

# Disable replicas temporarily for bulk ingestion
curl -X PUT -k -u elastic:PASSWORD https://localhost:9200/my-index-001/_settings \
  -H 'Content-Type: application/json' \
  -d '{ "settings": { "number_of_replicas": 0 } }'

# Re-enable after bulk load
curl -X PUT -k -u elastic:PASSWORD https://localhost:9200/my-index-001/_settings \
  -H 'Content-Type: application/json' \
  -d '{ "settings": { "number_of_replicas": 1 } }'

Next Steps

Configure Beats (Filebeat, Metricbeat) for data collection
Set up Log Shipping (Logstash or Fluentd)
Create custom dashboards and visualizations in Kibana
Implement alerting for critical thresholds
Plan capacity and scaling strategy

Elasticsearch Setup & Operations

On this page