Featured image of post Working with Elasticsearch in .NET Core

Working with Elasticsearch in .NET Core

Elasticsearch is an open-source search engine built on top of Apache Lucene™, the most advanced, high-performance, full-featured search engine library available today. While powerful, Lucene remains...

Elasticsearch Quick Start Guide

Introduction to Elasticsearch

Elasticsearch is an open-source search engine built on top of Apache Lucene™, the most advanced, high-performance, full-featured search engine library available today. While powerful, Lucene remains just a library. To leverage its capabilities fully, developers need Java expertise and direct integration into applications. Elasticsearch simplifies this process by:

  • Providing a distributed real-time document store with every field indexed and searchable
  • Offering a distributed analytics engine
  • Scaling to hundreds of servers and handling petabytes of structured/unstructured data

With official clients available for Java, .NET, PHP, Python, Ruby, Node.js, and more, Elasticsearch leads the enterprise search market per DB-Engines rankings, followed by Apache Solr (also Lucene-based).

Development Resources

Documentation

Downloads

https://www.elastic.co/cn/downloads/

Core APIs

API CategoryDocumentation Link
API ConventionsGuide
Document APIsDocs
Search APIsReference
Indices APIsGuide
cat APIsDocumentation
Cluster APIsReference
JavaScript APIClient Docs

Logstash Integration

ComponentDocumentation
Logstash ReferenceGuide
ConfigurationConfig Docs
Input PluginsList
Output PluginsReference
Filter PluginsDocs

Kibana DevTools Essentials

Keyboard Shortcuts

  • Ctrl+i: Auto-indent
  • Ctrl+Enter: Execute query
  • Down Arrow: Open auto-complete menu
  • Enter/Tab: Select auto-complete suggestion
  • Esc: Close suggestion menu

Append ?pretty=true to any query string for formatted JSON responses.

Common Commands

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# Check disk allocation
GET _cat/allocation?v

# List all indices
GET _cat/indices

# Sort indices by document count
GET _cat/indices?s=docs.count:desc
GET _cat/indices?v&s=index

# List cluster nodes
GET _cat/nodes

# Cluster health status
GET _cluster/health?pretty=true
GET _cat/indices/*?v&s=index

# Get shard information
GET logs/_search_shards

Cluster Health Check

1

curl -XGET 'http://<host>:9200/_cluster/health?pretty'

Sample Response:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

{
  "cluster_name": "es-cluster",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 3,
  "number_of_data_nodes": 3,
  "active_primary_shards": 1,
  "active_shards": 2,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100.0
}

CRUD Operations

Search Queries

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

POST logs/_search
{
  "query": {
    "range": {
      "createdAt": {
        "gt": "2020-04-25",
        "lt": "2020-04-27",
        "format": "yyyy-MM-dd"
      }
    }
  },
  "size": 0,
  "aggs": {
    "url_type_stats": {
      "terms": {
        "field": "urlType.keyword",
        "size": 2
      }
    }
  }
}

POST logs/_search
{
  "size": 0,
  "aggs": {
    "date_total_ClientIp": {
      "date_histogram": {
        "field": "createdAt",
        "interval": "quarter",
        "format": "yyyy-MM-dd",
        "extended_bounds": {
          "min": "2020-04-26 13:00:00",
          "max": "2020-04-26 14:00:00"
        }
      },
      "aggs": {
        "url_type_api": {
          "terms": {
            "field": "urlType.keyword",
            "size": 10
          }
        }
      }
    }
  }
}

Document Deletion

1
2
3
4
5

# Delete by query
POST logs/_delete_by_query {"query":{"match_all": {}}}

# Delete index
DELETE logs

Index Management

Creating Indices

Data migration essentially involves index re-creation. Note that reindexing doesn’t copy source index settings - configure mapping, shards, and replicas explicitly.

Migration Strategies

Cross-Cluster Reindexing

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

POST _reindex
{
  "source": {
    "remote": {
      "host": "http://source-host:9200",
      "username": "user",
      "password": "pass"
    },
    "index": "source_index",
    "query": {"match": {"test": "data"}}
  },
  "dest": {"index": "dest_index"}
}

Configure reindex.remote.whitelist in elasticsearch.yml for security.

Elasticsearch-Dump

Node.js-based migration tool:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

npm install elasticdump -g

# Transfer index components
elasticdump \
  --input=http://source:9200/my_index \
  --output=http://dest:9200/my_index \
  --type=analyzer

elasticdump \
  --input=http://source:9200/my_index \
  --output=http://dest:9200/my_index \
  --type=mapping

elasticdump \
  --input=http://source:9200/my_index \
  --output=http://dest:9200/my_index \
  --type=data

Deep Pagination

Overcome 10,000 document limit using search_after:

1
2
3
4
5
6
7
8
9

GET logs/_search
{
  "search_after": [1588042597000, "V363vnEBz1D1HVfYBb0V"],
  "size": 10,
  "sort": [
    {"createdAt": {"order": "desc"}},
    {"_id": {"order": "desc"}}
  ]
}

Installation

Docker Setup

1
2
3
4
5

# Elasticsearch
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.8.1

# Kibana
docker run -p 5601:5601 --link elasticsearch -e "elasticsearch.hosts=http://elasticsearch:9200" docker.elastic.co/kibana/kibana:7.8.1

.NET Integration

NuGet Packages

1
2

Install-Package NEST
Install-Package Swashbuckle.AspNetCore

Sample Entity

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

public class VisitLog
{
    public string Id { get; set; }
    public string UserAgent { get; set; }
    public string Method { get; set; }
    public string Url { get; set; }
    public string Referrer { get; set; }
    public string IpAddress { get; set; }
    public int Milliseconds { get; set; }
    public DateTimeOffset CreatedAt { get; set; } = DateTimeOffset.UtcNow;
}

Elasticsearch Provider

1
2
3
4
5

public class ElasticsearchProvider : IElasticsearchProvider
{
    public IElasticClient GetClient() => new ElasticClient(
        new ConnectionSettings(new Uri("http://localhost:9200")));
}

Repository Implementation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

public class VisitLogRepository : ElasticsearchRepositoryBase, IVisitLogRepository
{
    protected override string IndexName => "visitlogs";
    
    public async Task InsertAsync(VisitLog log) => 
        await Client.IndexAsync(log, x => x.Index(IndexName));
    
    public async Task<Tuple<int, IList<VisitLog>>> QueryAsync(int page, int limit)
    {
        var response = await Client.SearchAsync<VisitLog>(s => s
            .Index(IndexName)
            .From((page - 1) * limit)
            .Size(limit)
            .Sort(x => x.Descending(v => v.CreatedAt)));
        
        return Tuple.Create((int)response.Total, response.Documents.ToList());
    }
}

API Controller

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

[ApiController]
[Route("api/[controller]")]
public class VisitLogController : ControllerBase
{
    private readonly IVisitLogRepository _repo;

    [HttpGet]
    public async Task<IActionResult> Get(int page = 1, int limit = 10)
    {
        var result = await _repo.QueryAsync(page, limit);
        return Ok(new { total = result.Item1, items = result.Item2 });
    }

    [HttpPost]
    public async Task<IActionResult> Post([FromBody] VisitLog log)
    {
        await _repo.InsertAsync(log);
        return Ok("Created successfully");
    }
}

Verification

After implementation, use Kibana to validate data:

1
2

GET _cat/indices
GET visitlogs/_search

This guide provides foundational knowledge for working with Elasticsearch in .NET environments. For advanced query patterns and optimization techniques, refer to the official NEST Examples.