Rollover is a crucial action. It allows you to create a new write index when the current one meets a defined threshold. It makes your system more optimized and scalable. How does it work? Well, as always, let's observe its behavior through practice. Check out this project. It has an agent that executes GET _cat/thread_pool?v and GET /_tasks?pretty=true&human=true&detailed=true through a Python code. It parses the answers into JSON documents and sends them to index alias es-thread-pool or es-tasks. It requires an infrastructure in Elasticsearch to be created first. Execute the following command to create it:

./scripts/start-development.sh

Access the index templates page and use the credential elastic:elastic when it's complete. You'll see two custom index templates:

• es-tasks-index-template
• es-thread-pool-index-template

Now access the tab Data Streams. You'll see three system data streams created by Elasticsearch. Now execute the command:

docker compose run --rm -e TEST_COLLECT_METRICS=1 integration-tests python -m unittest tests.agents.test_elasticsearch.TestElasticsearch.test_collect_metrics

Two new data streams will appear, each one with one index:

If you click on the number of indices on es-tasks, you'll see its indices. In my case, it says there are 5 documents:

The rollover trigger is configured to happen when an index has at least 2 documents. After no more than 1 minute (know this), this is what happens:

A new index is created! It will occur every time a trigger criterion is matched. Keep in mind that this definition is for the sake of this post.

Don't need data streams? Change the flag enable_data_streams in the Terraform code and see how the rollover works when only indices are used without data streams.

I hope this may help you. See you! 😄