Fundamentals of Elasticsearch architecture and shards

Know the fundamentals first

Elasticsearch is a wonderful tool, teeming with useful features for many different industries. It's very easy tog get started and, for a while, enjoy incredible performance without the need for maintenance or tuning.

Elasticsearch seems like a golden bullet that has solved all your problems without needing to become a guru in yet another technology.

It doesn't take long before the situation changes and you begin asking questions like:

• Why does this query that used to take 250ms now take 5000ms?
• Why has the maximum indexing rate gone way down?
• What does this `doc_count_error_upper_bound` key mean and why isn't it zero?
• How many primary shards am I supposed to use for this index?
• I know the cluster is red but I'm still getting query results, so that's fine, right?

Some of these questions are easy to answer, others are complicated and highly specific to your environment. The answer to each of them requires understanding what a shard is, and how it plays a part in an Elasticsearch cluster.

It's a common knowledge gap

I've spoken with a lot of people who work with Elasticsearch and this fundamental understanding of shards is a common gap in knowledge.

When I've talked those people through the basics, they are much more aware of the effect of different shard configurations and can understand why their issues are happening. It can be a real lightbulb moment and gives people the confidence to make better choices and learn more about Elasticsearch, and The Elastic Stack as a whole.

This course will start to fill that gap

I built this course to have a resource I can refer people to when I sense that a grasp of how shards will help them.

It's not a complete guide but it will help give an appreciation for how a cluster works, and how shards are the fundamental building blocks.

The knowledge you pick up here will also make it easier to know other pitfalls in some parts of Elasticsearch.

I hope you find it useful! I'd love to hear feedback, so please get in touch if there's anything you'd like to share.